Perils of URLClassloader
Its a very common programming technique to create any number of URLClassLoader instances repeatedly in order to load the new implementation of the classes/resources from the same location with the help of new class loader instances.
In fact, some well-known tools like plexus-compiler jar makes heavy usage of IsolatedClassLoader to implement the above concept.
Lets focus on the actual problem.
We have a large maven project which has 200 jars in maven repo.
Any code change in source file fires – MavenBuilder …..
- which invokes JavacCompiler (plexus-compiler jar)
- in turn creates IsolatedClassLoader cl = new IsolatedClassLoader() [extends URLClassloader] ..
- adds the list of 200 jars in urlClassPath - cl.addURL(URL)
- loads javac.Main along with all jars in classpath and reflectively fires compilation.
So for N save-operations, N instances of IsolatedClassloaders are created.
In principle, once the application clears all references to a loader object, the garbage collector and finalization mechanisms will eventually ensure that all resources (such as the JarFile objects) are released and closed.
But in reality, the application goes OutOfMemory !
We used optimal concurrent gc strategy - Xgcpolicy:optavgpause.
But still the app running out of memory very quickly.
Then Heap dump analysis showed there are some 100 instances (Uuh !) of isolatedClassloaders each of which is holding onto unclaimed ZipFileIndexEntries(.. guess what .. indexEntry for all the jars loaded on each instance of class laoder..)
Well ! looks like.. since a new URL Class Loader is created before closing the resources in previous loader, GC is confused and does not reclaim the previous class loader ! This causes problems for applications which need to be able to GC in a predictable and timely fashion. It is a particular problem on Windows, because open files cannot be deleted or replaced.
So are we missing something trivial here ? Is plexus-jar doing something wrong ?
A BIG Emphatic YES !
We should always use sun.misc.ClassLoaderUtil to release an url class loader and file resources held therein.
Lets now fix the issue.
org.codehaus.plexus.compiler.javac.JavacCompiler#compileInProcess(..) {
IsolatedClassLoader loader= new IsolatedClassLoader();
loader.addURL(jarListoURI().toURL())
c = loader.loadClass( “com.sun.tools.javac.Main” );
ok = (Integer) compile.invoke(args)
//Now that you are done with compilation, get rid of loader ..
*** loader.close () Or sun.misc.ClassLoaderUtil.releaseLoader(loader)
////// -> allows graceful release of loader and all resources
*** Thankfully JDK 7 URlClassloader has implemented close() method to give the caller a chance to invalidate the loader, so that no new classes can be loaded from it. It also closes any JAR files that were opened by the loader. This allows the application to properly delete or replace these files and, gracefully create new loaders using new implementations.
The bottom line is wherever we create seemingly harmless URlCLassloader in a repeated manner (say to generate / compile code) holding onto a good many jars in url classpath; we should always close the loader instance.
The most advanced software factory!
Software Development has come to a mature state with standard frameworks and technologies that could be adopted to solve a class of business problems and or with specified set of architectural stacks. With most recent software development processes like SCRUM, extreme programming, and the customers are closely working with the team to get the software delivered to customers more effectively. The software development and delivery has become more mature and it is akin to manufacturing industry where assembly lines rollout vehicles continuously, except in software development, the assembly line is the sprint that delivers workable, testable and usable software.
In such a scenario, what are the challenges? The line of business managers are constantly presented with challenge of choosing the right partner, right vendor to source. The process of evaluating vendors experience with appropriate domain and technical stacks is a challenge. Other challenges include choosing an architecture that suits him based on his short and long terms goals.
There are numerous options to source work with. There are biggest like Infosys, TCS who can offer their vast experience, stability and experience. There are small players who can get to the goal in shortest and most economically way. They are also niche players who can get the software out with right parameters. The driving factors for choosing a vendor political, administrative, technical, and geographical and mostly the biggest factor is the financial viability as well.
Once the vendor is selected, you will start working with the vendor the exact process, set deadlines and quality goals and continuously monitor with metrics that could validate your choice of the vendor. Unfortunately, there is no standard for metrics and most of the time, the experience of the managers with different vendors serve as the most important metric in defining the relationship.
With more than 100 years of accumulated software development experience and a decade of experience in developing marketable products, and with four successful products under belt, we have a case to prove that, we are the most advanced factory producing software in the world. We have mastered the art of producing software. Be it be the user interfaces, be it the scalable designs we come up, be it with the flexibility of the software, be it the perspective and attitude we work with, be it with the code quality, we bring in right value.
We will help you focus on your key goals to get your product right and in time to market. Instead of worrying about development, we help you to focus on the market. This results in taking the huge burden off of your shoulders, which will allow you to focus more on business rather than day to day headaches of software development. You should not be worried about technology, execution and implementation. With the strong, agile light weight process that is customized for each customer we drive in value to right from day one till the end.
If you have an idea and want to take the market, we can help you take it to the market in the shorted passible time frame. Since, we are saying, we are a factory, we have mastered the art of producing software. We can produce software at faster, better and competitive.
You name the technology that you want to work with. Our factory has the skill and expertise to weave your product. You want to work with Social media or ERP systems, health care or advanced mining applications, you name it, we can get you there in short period and in time.
Over the years we have not only mastered the art of producing software, we have built a strong team of technology evangelists who can help you build solution with most of the latest state of the art technology stacks we have available today.
Imaginea is the destiny for most of the vendors who understands the maturity of software development, in terms of technical frameworks, products, time to market etc. We are the right partner of choice with right people, right size, and right people and with the right mind set.
A successful product development hinges on the effective sharing of ideas, leveraging organizational knowledge and ideas by its talented workforce. Effective knowledge sharing within teams takes substantial time. Imaginea is built on the fundamentals of sharing, mentoring and coaching people to become technology evangelists, entrepreneurs and business leaders.
If you are not convinced, give a call to our marketing or send us an email or talk to some of our customers.
Successful Product Management in an Agile World
Having a Right product strategy is a critical factor in making sure the desired product comes out of agile teams. As a leader in providing product development services to ISV’s at Imaginea, we have over the period of time understood what successful product managers do to roll out world class products. Some of us have proven experience in sharing these experiences to value add to our customers. We have closely watched some of the best product managers that we have worked with, and have fine tuned the art of managing and delivering products. Our own products are a reflection of some of these practices am going to speak about.
I have been trying to understand the Agile practices for quiet some time now. Having closely examined few projects that have been adopting agile practices, i made certain notes and observations. In my free time i also interviewed few successful agile practitioners, professionals and scrum masters. I have also spoken to some senior product development engineers who have shared their experiences. Here are some thoughts:
Background/ Context
Another organization that i knew practices agile methods to roll out the product it has in mind. Recently I spoke with a team of very frustrated Product development engineers there who were using Agile methods. They were not so happy for a simple reason that, they were working for the past one year trying to finish the features they were responsible for.But the hard part was that the product manager really had no idea where they were moving or what they were really trying to accomplish.
That was when i realized that i should talk to the product manager (being a product manager my self previously it was easy for me to reach to him). Speaking to him, i realized he is also the product owner- as Scrum calls it.He explained that he thought the whole idea of Agile methods was to be nimble, remain flexible and agile. But he was not convinced of the fact that more important was to be sure about the long-term direction of the product.I started realizing that creating an effective product strategy may have become an unintended casualty of the move to Agile methods.
I now strongly believe we need to understand what a right product strategy is, why is it very critical for a successful product launch and its complete alignment with the Agile methods.
What is a product strategy
Loosely put, Product strategy is the vision you wish to accomplish say in 2-4 years. It is for sure not a specification or a document. It is certainly a vision the leader or the product owner must be able to rally the teams with. If this objective is met correctly half the job is done.
Product Strategy communication
Modern day product managers use a lot of collaborative technologies to communicate the strategy to their teams. It could be wiki’s, e2.0 products, presentations or the same old style documents. The critical factor here is the communication has to be clear and inspiring. It could be a choice of presenting in person or through a shared desktop presentation which will not be a differentiating factor.
What must a Right Product Strategy Address
a) The difference that the product aims to make to its intended customer base.
b) Existing problems that the customer has and how they are solved using the product in making.
c) What will be those WoW factors that the users would resonate with in using the product
d) Essentially product strategy is a link between the corporate business strategy and the road map that the product wishes to traverse. Product strategy thus must enable the achievement of business strategy and road map must help you reach there step by step.
e)A good product strategy is one of the most important things a product manager should look to have. It’s not easy but without it you have little hope of actually ending up with something worthwhile.
f)Make sure your executive teams reviews, approves and appreciates the product strategy. Do not assume that it will come by itself from top-down. It will never happen.
g) Only if you’re in a start up with a founder serving as the product visionary there is a chance that the product strategy gets pushed from top-down. Otherwise product manager must hold the forte and communicate the same to the management.
h) It is essential to understand that the product strategy should not bind/ restrict into any particular features or sequencing. Features and their order of implementation are represented as part of the backlog.
i) However as an alert and efficient product manager to make sure you are on top of the schedule, You can, and absolutely should, adjust the roadmap constantly
based on what you learn from your users, the market, your analysis, trends, reports, and
the ever-changing technologies that the product is being built upon.
j)Majorly impacting the product manager’s decision point is the trade-off’s that will arise during the course of the implementation journey. A product manager here must wear the hat of the customer first and engineering next. That will also ensure he understands what the marketing and sales would prefer going into to the field than pure play engineering.
k)Here i believe that Agile methods, applied properly, will help you make
your product strategy a reality. Also it would ensure its significantly swifter than the traditional methods.It gives you the flexibility to turn around in 2 weeks per sprint if some thing is really worth changing as part of the trade-offs you encountered.
Conclusion
Having a right product strategy, implementing the strategy in alignment with the business plan and fine tuning the road-map will be the key to successful product management. product backlogs have to be effectively managed to ensure the right return on investments are in place from your agile teams. This way your engineers, your management, your partners in delivery will all be equally participative in getting the right product out.