Single Sign On using SAML
SAML (Security Assertion Markup Language) is an XML-based framework for communicating user authentication, entitlement and attribute information between different organizations. SAML is developed by OASIS (Organization for the Advancement of Structured Information Standards). SAML abstracts authentication mechanism so that it is not tied to any vendor specific implementations.
SAML Participants
In Web SSO user authenticates to one web site and then without additional authentication user is able to access a protected resource at another site.
Identity Provider:
Identity Provider is an entity that authenticates users during web SSO. Identity Provider creates maintains and manages user’s identity information.
Service Provider:
Service provider is an entity that provides services. Service provider relies on a trusted identity provider for user authentication and authorization. Service provider is also called as Relying Party.
SAML addresses following limitations
- Most of the existing products which support SingleSignOn use browser cookies to maintain state, so that re-authentication is not required. Browser cookies are not transferred between domains. For providing the cross domain SingleSignOn, SSO products use different techniques.
- When SSO products which have proprietary mechanism are used to provide the cross domain SSO, all the partners need to use same SSO product, which may not be possible in all cases.
Since SAML is an XML-based framework, it has extensibility which makes it a very flexible standard. Interoperability also gives huge advantage over SSO products which has proprietary mechanism.
SAML2 Use Cases
Identity provider initiated SSO:
In an IDP initiated use case, Instead of visiting SP directly, the user accesses the IDP site and then clicks on one of the links to gain access to the protected resource on service provider. This triggers creation of a SAML assertion which is sent to the service provider.
Service provider initiated SSO:
In SP initiated SSO, the user attempts to access a protected resource on SP. As user does not have a session on SP, User is sent to the IDP for login and IDP provides a SAML assertion with the user identity back to the SP.
Service provider initiated Single Logout:
In SP initiated log out, SP destroys the local session and then sends a request to IDP requesting log out on user session.
Identity provider initiated Single Logout:
In this use case, log out process is initiated by the user visiting the IDP.
Identity provider initiated Single Sign On
Identity provider is configured with the links that refer to the service provider; these links actually refer to the IDP’s SSO service. When a link is clicked, IDP generates SAML assertion which is sent to the remote service provider. SAML assertion contains user authentication and authorization information.
1) User tries to access the IDP
2) User is asked to provide credentials.
3) User provides valid credentials.
4) User access links on the IDP to request access to the remote service provider.
5) IDP SSO service creates a SAML assertion which contains user information. SAML Assertion is sent to service provider using HTTP POST.
6) Service provider’s Assertion Consumer Service get the SAML assertion from HTTP POST request and validates the digital signature on the SAML assertion. Assertion content is processed in order to create a session for the user in SP. If Relay State information is provided then SP determines desired application resource URL and sends HTTP redirect to the browser directing to access the requested resource.
In order to configure SSO, Service Provider needs to be registered with the IDP. To accomplish this, the SAML specification defines a format for “SAML Metadata” which tells the IDP about service provider’s Assertion Consumer Service. Metadata file can help the identity provider understand what the service provider need in the SAML assertion.
Example MetaData:
<EntityDescriptor entityID=”http://localhost:8081/sso“>
<SPSSODescriptor protocolSupportEnumeration=”urn:oasis:names:tc:SAML:2.0:protocol”> <NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</NameIDFormat> <NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</NameIDFormat> <NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</NameIDFormat> <NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified</NameIDFormat> <AssertionConsumerService index=”0″ isDefault=”true” Binding=”urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST” Location=”http://localhost:8081/assertion_consumer_service.jsp“/>
</SPSSODescriptor>
<Organization>
<OrganizationName xml:lang=”en”>Pramati</OrganizationName> <OrganizationDisplayNamexml:lang=”en”>Pramati</OrganizationDisplayName>
<OrganizationURL xml:lang=”en”>www.pramati.com</OrganizationURL>
</Organization>
</EntityDescriptor>
Example SAML Assertion with subject
<saml:Assertion>
<saml:Issuer>https://pramati.com/sso/saml2</saml:Issuer>
<saml:Subject>
<saml:NameID Format=”urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress”> venkat@imaginea.com
</saml:NameID>
</saml:Subject>
</saml:Assertion>
Access Rights Testing using BURP
Overview:
Any web application can have different levels of access permissions for various pages of the application. The levels of access rights / authorizations can be Admin users, End users, guest user’s etc., End users should not be able to access the administration pages and these access restrictions are usually implemented at JSP level.
The annotation of helper methods and classes on the backend can be overlooked by depending on the access restrictions provided by JSPs and so the problem is that many of these Helper methods, and the Service methods they call, don’t enforce their own security; they rely on the JSPs and the access rights XML to enforce their own security.
The result is that a user can call any form-action method of a helper from any JSP that can access that helper, even if different permissions would normally be required.
The fix is adding a set of annotations which will be added to helper methods and classes. These annotations mark the authorization required to perform a given form-action, before invoking the form-action method.
Tools:
- Burp Proxy (used the intercepting capability of burp)
- JRE1.5 or later
About BURP:
BURP is an open source security testing tool used to identify security vulnerabilities in web applications. It has various features for testing. These include proxy, spider, intruder, repeater, sequencer, decoder and comparer. The proxy feature of burp was used to test access right vulnerabilities in the application.
Burp Proxy: Using Burp proxy, one can intercept the traffic between the browser and target application. This option works in similar fashion to the man-in-the-middle attack vector. After setting up burp as a proxy to intercept the requests from the browser, switch the intercept mode “on” in the suite. When a request is intercepted by burp, user has two options, Forward and Drop. The Forward option allows the user to send the packets from the source IP to the destination IP. The Drop option allows the user to drop the packet if you feel it does not need analysis.
Test Approach:
Burp is used to test if security has been implemented correctly
- For each Helper class, capture a request and edit it to call specific form-actions that can be called from various types of users. (Not logged in users, End users, Admin Users, and guest users etc.,)
- An end user should only have access to the form-actions they are intended to have access to (End user authorization, and ‘no authorization required’). Security verification will throw an error when end users try to access a method they do not have access to (admin authorization, Internal Admin authorization, and not a form-action).
- Admin users will have access to the form-actions they are supposed to. Admin user, if they choose to, can access admin methods through the buy-side (by capturing and editing requests) and security will not block the request. Admin users cannot access Super Admin methods.
- Actions that are supposed to run (valid security token) run properly
- Also test XSS vulnerabilities in the input fields by passing script into field values
Sample requests from BURP:
User has to modify the helper and use the helper he wants to test (for advanced testing)
Form action parameters that have to be modified (These are the methods under test)
Example:
Consider a case where a helper class, say H1 has two methods M1 and M2. M1 should be accessible to all users (admin and normal users) whereas M2 should be accessible only to admin users. Consider two JSPs, say JSP1 is accessible to all users and it can access the helper class H1 and JSP2 is accessible only to admin users and it can access the helper H1. In this case, if the methods in the helper are not annotated properly, normal users (who don’t have access to M2 in H1) with some advanced knowledge can access the admin method M2 by accessing the jsp JSP1 and modifying the form action parameters in the request sent from JSP1 to access the admin methods of H1.
Benefits:
Testing all the helper classes led to proper use of annotations for all the helper methods and classes which were overlooked earlier at the mercy of security implemented at JSP level. Many XSS vulnerabilities were also observed and were fixed. However, for testing access related vulnerabilities using burp, testers need to know which JSP can access both admin and buy-side related helpers. These testing methods are highly useful for applications that have different access levels for different users. Using these testing methods can help testers identify major security flaws in the application well ahead in time and save a lot of time and resources, which wouldn’t have been possible through testing from UI.
Message Forwarding and Surrogate Objects in Objective-C
Recently, I came across a tricky problem in one of the projects that I am working on. We are using a framework which maps Objective-C classes to database columns. This framework will be accessing properties of the mapped class while saving and retrieving data. I have couple of classes, primary data-model class and a secondary data-model class which is used as property of the main class. The problem is to access all the properties of the secondary class through primary class.
Straightforward solution is to implement the getters and setters in the primary class to get and set the properties of the secondary class. This solution will solve the problem, but it is not very elegant and scalable. Looking for other solutions, I found the Objective-C concepts like Messaging and Message Forwarding which can be used for a scalable solution. Before providing the solution, I will explain in brief about the Messaging and Message Forwarding of Objective-C
Messaging
The Objective-C model of object-oriented programming is based on message passing to object instances. In Objective-C one does not simply call a method; one sends a message. This is unlike the Simula-style programming model used by C++. The difference between these two concepts is in how the code referenced by the method or message name is executed. In a Simula-style language, the method name is in most cases bound to a section of code in the target class by the compiler. In Smalltalk and Objective-C, the target of a message is resolved at runtime, with the receiving object itself interpreting the message.
The key to messaging lies in the structures that the compiler builds for each class and object. Every class structure includes these two essential elements:
- A pointer to the superclass.
- A class dispatch table. This table has entries that associate method selectors with the class-specific addresses of the methods they identify.
When a message is sent to an object, the messaging function follows the object’s isa pointer to the class structure where it looks up the method selector in the dispatch table. If it can’t find the selector there, objc_msgSend follows the pointer to the superclass and tries to find the selector in its dispatch table. Successive failures cause objc_msgSend to climb the class hierarchy until it reaches the NSObject class. Once it locates the selector, the function calls the method entered in the table and passes it the receiving object’s data structure. This is the way that method implementations are chosen at runtime—or, in the jargon of object-oriented programming, that methods are dynamically bound to messages.
Message Forwarding
Sending a message to an object that does not handle that message is an error. However, before announcing the error, the runtime system gives the receiving object a second chance to handle the message.
You can implement a forwardInvocation: method to give a default response to the message, or to avoid the error in some other way. As its name implies, forwardInvocation: is commonly used to forward the message to another object.
Forwarding and Inheritance
Forwarding mimics inheritance, and can be used to lend some of the effects of multiple inheritance to Objective-C programs. An object that responds to a message by forwarding it appears to borrow or “inherit” a method implementation defined in another class. Forwarding not only mimics multiple inheritance, it also makes it possible to develop lightweight objects that represent or “cover” more substantial objects. Although forwarding mimics inheritance, the NSObject class never confuses the two. Methods like respondsToSelector: and isKindOfClass: look only at the inheritance hierarchy, never at the forwarding chain.
Surrogates
The surrogate stands in for the other object and funnels messages to it. If we use forwarding to set up a surrogate object or to extend the capabilities of a class, the forwarding mechanism should probably be as transparent as inheritance. If you want your objects to act as if they truly inherited the behavior of the objects they forward messages to, you’ll need to re-implement the respondsToSelector: and isKindOfClass: methods to include your forwarding algorithm. In addition the instancesRespondToSelector: method should also mirror the forwarding algorithm. If protocols are used, the conformsToProtocol: method should likewise be added to the list. Similarly, if an object forwards any remote messages it receives, it should have a version of methodSignatureForSelector: that can return accurate descriptions of the methods that ultimately respond to the forwarded messages.
As we have enough info about the Messaging and Message forwarding concepts, lets discuss the solution. In the project I am currently working on , there is a search feature. User can search for results using some criteria. This criteria consists of primary and advanced criteria, and datamodel classes are in place for both. Primary datamodel will contain secondary datamodel as a property. There is a framework which takes care of mapping database tables to specified datamodels. To save or retrieve data, the framework will be sending message to the mapped datamodel class. This framework doesn’t support nested mappings. Primary datamodel class is mapped to the database table, hence the framework will be expecting the columns to be mapped to the properties and will be sending setter and getter messages to the primary datamodel class. Solution is to make the advanced data model a surrogate of the primary data model. In the primary datamodel class, by overriding the following methods, the secondary datamodel is set up as the surrogate for the primary datamodel class.
- (NSMethodSignature*)methodSignatureForSelector:(SEL)selector
{
NSMethodSignature* signature = [super methodSignatureForSelector:selector];
if (!signature) {
signature = [surrogate instanceMethodSignatureForSelector:selector];
}
return signature;
}
- (void)forwardInvocation:(NSInvocation *)anInvocation
{
if ([surrogate respondsToSelector:[anInvocation selector]])
[anInvocation invokeWithTarget:surrogate];
else
[super forwardInvocation:anInvocation];
}
If I had used the initial approach of manually adding setters and getters, I would have ended up writing as many methods as twice the number of properties and had to make sure all the method signatures mimics the advanced criteria properties. Any changes to the name of the properties is a good entry point for regressin and code maintenance is going to be a mess. Other approach is to maintain a single datamodel which will represent all the criteria which will create other issues as the requirements for iPhone and iPad are different. Message forwarding approach reduced the amount of manual effort required and will be scalable even if there are changes to the datamodels.
It’s time to become “Responsive “
Responsive design architecture as the name suggests is an interactive design principle achieved by being sensible to the environment and responding accordingly to give the user/ customer with an optimum experience. Responsive design is adopted across multiple fields like Architecture, Art, and Electronics. For example we have Smart Air Conditions adjusting the temperature of the room depending on the number of occupants in the room and living spaces which can reform themselves to enhance the experience of the occupants. Adopting the same methodology to web Ethan Marcotte has coined Responsive Web Design.
What is Responsive Web Design (RWD)
With an exponential increase in usage of mobile devices to surf web sites the existing web sites are expected to scale to mobile & tablet browsers. In order to make a the web site accessible in multiple mediums the designers of the web site come up with different version of the same site to support mobile . This would lead to creating multiple disconnected designs to support multiple devices for the same web site. RWD tries to avoid creating various versions of the same web site by presenting the content in most accessible manner for any viewport that access it. In short a design that can adapt to the constraints of the browser window or device that renders it, creating a design that almost responds to the user’s need.
When should we adopt RWD
RWD is one of the simplest and quickest ways to make a website work on a lot of devices. RWD should be adapted when the web site content remains the same be it any device and the only variation is the way it is being displayed. For example the small screens put elements of less importance beneath the main content or as a worst-case scenario, hide them altogether. Maybe alter navigation buttons to accommodate finger presses, rather than only offering a usable experience to those able to proffer a precise mouse click. Typography should also be scaled for the sake of readability, allowing text to be read without swipes from side to side.
There are scenarios where RWD is not the right solution if the mobile site totally serves a different purpose than the existing web site and it needs a different content, design, and interaction based upon the device, location, connection speed and other variables like the technical capabilities of the device.
How to become Responsive
In order to make a site responsive the following are the key aspects to be considered during web site design in brief.
1. A flexible, grid-based layout.
To achieve flexible grid layout all the pixel based layouts are to be replaced with the percentage based ones. The margins, paddings and fonts are also to be percentage based.
2. Flexible images and media
Images need to scale according to the size of the viewport .This can be achieved either through scaling the size of the image or cropping the image as the functionality demands.
3. Media queries (CSS3 specification )
With CSS3 media queries different styles for different viewports can be conditionally loaded and hence creating an optimized look for that particular view port.
Will come up with a working example in my next post regarding how to make a site responsive using the aspects mentioned above.
Popular Sites which adapted RWD
RWD has been adopted by many sites since its inception. Some of the popular ones which adopted this design principle are Disney, Microsoft, StarBucks. Sample link
http://www.microsoft.com/en-us/default.aspx
Extracting topics of interests using blog data in Amazon’s Common-Crawl corpus
Overview
This project aims at profiling blogger interests correlated with their demographics. Amazon’s Common-Crawl corpus was used for this purpose. The crawled data corresponding to the blogger profile web-pages(Sample page) was used as the dataset for this analysis.
The selective download of the required dataset was made possible by the Common Crawl URL Index by Scott Robertson. About 8000 blogger profile pages(surprisingly low!) were found in the corpus using the URL index. Part of the reason for this low number is that the URL index at this time has been generated only for the half of 81TB amazon corpus.
Check out the project at GitHub
Some finds from this exercise were
1) 25% of blogger from Bangalore write about Music(Not so surprising as it is the rock capital of india)
2) NY, SF and Toronto account for 40% of the bloggers.
3) San Francisco and Dallas, each 17% account for the most bloggers with interest in Politics
4) San Francisco,Bangalore and Vancouver are among the top cities with bloggers whose interest is travel
The technology stack of the project comprises of
- Python script to facilitate downloading the selective chunks of ARC files in Common-Crawl corpus. This uses the JSON generated from the URL Search web-application API, when search for all domains with blogger.com/profile as prefix.
- Jsoup for crawling downloaded blogger profile web-pages
- Maui to extract topics from raw text. This was used in conjunction with AGROVOC vocabulary.
- MySQL database is used to dump the rough results of the crawl of the blogger profiles for further analysis.
- JFreeChart is used to display charts.
Extracted Dimensions : City,State,Country,Topic of Interest, Gender,Occupation
Extracted Measure : No of bloggers
The aforementioned dimensions were used in the generation of the reports. The sample reports can be found in the SnapShots folder in the GitHub project.
Thought process behind the project
Reason to use URL Index
Cost aversion was a factor which lead to the usage of the URL index, thus allowing for all the computation to be done in my local machine.
Reason to use Maui
Maui is a topic extractor having the ability to extract topic terms from text, even those which are not actually present in the text
For example, it could give me “Politics” as a topic, even though the term “politics” or any of its roots not being actually mentioned in the text.
Maui consumes RDF based vocabularies for topic extraction.I used the latest version of AGROVOC vocabulary as it contains a broad array of topics.
Reason to use blogger profile web-pages
Since the purpose was to analyze blogger interests based on demographical information, these profile pages served as a rich source of that information.It was possible to infer about the topics which are of interest to the blogger, without actually having to crawl his entire blog.As this was just a Proof of Concept(PoC) project, this minimized the computation cost as everything was done locally, without the need of EC2.
Future
The blogger profile page also contains two sections called “My Blogs” and “Blogs I Follow”. The blog pages corresponding to these could also be downloaded and crawled for topic extraction. This job would be best done using the EC2 infrastructure as the data size could be exponentially greater. The python script could be modified to use the actual URL index located here to find out if we missed any data. But this would also require EC2 infrastructure as the index size is huge(250 GB!).
Disassembling JVM – Execution Model
The role of JVM’s execution engine can be compared to that of a microprocessor. Its duty is to execute the instructions fed to it in some acceptable format. This is achieved by either interpreting the bytecode instructions or compiling them to native instructions as the need may be. JVM’s execution engine, in a way, behaves like an assembler, so to say. Those who are familiar with assembly language can easily relate JVM’s instruction set to that of modern microprocessors. In this final instalment of ‘Disassembling JVM’ series we will have a look at how JVM executes the contents of a class file.
The Instruction Set
The executable portions of a class file are its methods, be them user-written or synthetic. After compilation, the instructions pertaining to methods are stored as Code attributes in the attribute info section of the .class file. During classloading, these get mapped to method area. When a method is invoked, these instructions are executed one by one sequentially.
JVM’s instruction consists of a one-byte opcode and zero or more operands. Due to one-byte limitation, its instruction set has only 256 instructions in all. The opcode specifies the operation to be formed like, add, subtract, object instantiation, method invocation etc. and operands specify the data to be used for the operation. Not every opcode has operands associated with them; hence a bytecode instruction can be of variable length. It can be considered to have the following format.
Most of the JVM instructions are typed, i.e., we can infer the type of operands from the opcode itself. For example, iadd means add two integers and fmul means add two floating-point numbers. Due to one-byte limitation, there are no equivalent operations for all types, especially for byte, char and boolean types. They are mostly handled using integer type opcodes. Also, most of the instructions manipulate operand stack and local variable table. There are shortcut opcodes like iconst_0, iconst_1, iconst_2, iconst_m1 etc. that load frequently used constants 0, 1, 2, -1 etc. onto the stack and iload_0, iload_1, iload_2 etc. that load values at frequently used local variable indices onto the stack.
The instruction set can be broadly classified into ten categories:
- Load and store: These are used to transfer values between local variable table and operand stack. E.g.: iload, istore, bipush, iconst_x, ldc
- Arithmetic and logic: These are used to perform integer and floating-point arithmetic and logic operations. E.g.: iadd, iinc, fdiv, dmul, iand,ineg, dcmpl
- Type conversion: They are used for numeric type conversion, especially to widen and narrow byte, short and char types as we do not have many instructions for them. E.g.: i2l, i2f, i2b, i2c
- Object manipulation: They are used to create and manipulate class and array objects. E.g.: new, newarray, getfield, putfield, aaload, aastore, arraylength, getstatic, pustatic, instanceof, checkcast
- Operand stack management: They are used to manipulate the operand stack. E.g.: pop, push, dup, swap
- Control transfer: They are used to transfer control conditionally or unconditionally. E.g.: ifeq, ifne, ifnull, ifnonnull, if_icmpeq, tableswitch, lookupswitch, goto, jsr, ret
- Method invocation: They are used to invoke methods and return from them. E.g.: invokevirtual, invokespecial, invokestatic, invokeinterface, return, ireturn, freturn
- Exception handling: An exception can be thrown using athrow instruction. Runtime exceptions are thrown directly by the JVM.
- Synchronization: They are used to manipulate object monitor for synchronizing methods and instruction blocks. Two operations available for this are monitorenter and monitorexit, which are called when the synchronized block is entered and exited.
- Reserved: These are opcodes reserved for internal use and for future extension of the virtual machine and may not appear in method bytecode. They are impdep1, impdep2 and breakpoint.
It would be difficult to explain each and every instruction in this article. For a detailed explanation, please refer the specification. Let’s look at a sample program and try to understand its bytecode instructions before concluding.
The equivalent bytecode disassembled using javap tool is given below, with unnecessary details removed. You would find that the instructions are pretty much straight-forward except for some cryptic numbers prefixed with ‘#’. They are nothing but references to constant pool of the class. It is not shown here for the sake of brevity.
A Word on Multi-threading
When JVM is invoked, it runs as a new process on the host operating system and forks a new non-daemon thread to execute the main method class. This JVM instance would continue to live till the last non-daemon thread exits or till Runtime.exit() is called. During its lifetime it might fork separate threads internally for its own house-keeping activities like garbage collection, management and monitoring etc… So we could visualize a live JVM to be consisting of a group of threads which are trying to execute some tasks – user-defined or system-defined.
So how does JVM handle multiple threads? Does it emulate threads or actually create native threads? If you have read the previous articles, the answer would be obvious – it’s implementation specific! Matter-of-factly, the specification doesn’t even talk about how they should be implemented. Prior to Java 1.1, multi-threading was implemented using green threads. These days, however, implementations settle for green (emulated) threads or native threads or sometimes a combination of both, depending on the environment or user configuration. Green threads are particularly useful when native operating system doesn’t support multi-threading. But they come with additional burden of implementing scheduling, memory management etc. and hence may not be suitable for performance intensive scenarios. The whole application may stall if one of them is waiting on I/O, virtually bringing down the throughput to zero. Native threads on the other hand are efficient and can take advantage of software and hardware optimizations of the host system.
We have only covered the tip of the iceberg here. Most of the information were taken from the JVM 7 specification. Developers should be encouraged to go through it to have a better understanding of JVM and appreciate the efforts that have gone in to develop such a superb piece of software.
Disassembling JVM – Linking Model
In case of natively compiled languages, linking is the last step of generating executable from source files. Linker takes in object files, output by compiler, and builds an executable file which could be further run without any dependencies. This approach of statically linking related code had considerable impact on memory and limited reusability of shared code. With the introduction of dynamic linkers, this problem was overcome by importing shared code and combining them with the running process. In native applications, dynamic loading and linking of shared libraries are handled by the operating system itself, sometimes offering support for different versions of the same library as well.
Dynamic loading and linking, though done differently, happen almost implicitly in Java – unless one starts seeing ClassNotFoundException or NoClassDefFoundError that is. In the context of Java, loading is the process of finding and creating a class/interface from its binary, and linking is the process of integrating it with the runtime state of JVM. These essentially involve mapping a class/interface to JVM memory – adding appropriate type definition to method area and creating a new instance of Class for it on the heap. (Linking should not be confused with the process of binding that happens when a native method is invoked. Binding is handled by native interface module and won’t be discussed in this series.)
Since most of the Java developers would be familiar with classloading principles, instead of explaining the entire topic once again, we would be covering only the essential details here.
Types of Classloaders
A class/interface is loaded into JVM on its first use and is usually unloaded by the garbage collector when it is no longer required. Part of JVM that does loading and linking of classes/interfaces is undoubtedly called the classloader and is abstracted by ClassLoader class. Developers can extend ClassLoader to create specialized classloaders that could load classes from different sources or possibly instrument them before use.
In a broad sense we can classify classloaders into bootstrap/primordial classloader and user-defined classloaders.
- Bootstrap classloader: This classloader is responsible for loading core java classes (those located in jre/lib or as defined by bootclasspath argument). It is natively written and hence cannot be extended.
- User-defined classloaders: They load other application specific classes and can be identified as subclasses of ClassLoader. This category includes system/application classloader that load classes found in the classpath (or as defined by java.class.path property) and custom classloaders written by developers.
With the introduction of extension mechanism in 1.2, most implementations also support a third type of classloaders called extension classloader. This classloader is used to load optional packages (those located in jre/lib/ext or as defined by java.ext.dirs property) that add functionality to the core platform.
Classloading Mechanism
The standard classloading mechanism used by JVM (since 1.2) is called the parent-delegation/parent-first model. In this arrangement, a classloader always has a parent, either a user-defined classloader or the bootstrap classloader. (Remember, this is a delegation hierarchy and not based on inheritance.) When a class/interface is to be loaded, the active classloader checks if it has been already loaded by it, if it has not, the request is delegated to the parent classloader. If the parent classloader cannot load the type, it would delegate it further to its own parent until the request reaches the bootstrap classloader. If the bootstrap classloader also cannot load the type, the initiating classloader tries to load it on its own or throws ClassNotFoundException. Again, applications can override this pattern and follow their own classloading logic if functionality demands so. A good example for this could be web application classloaders that follow a parent-last model in order to allow libraries in WEB-INF to take precedence over classpath libraries.
Inside JVM, a class/interface is jointly identified by its fully qualified name and classloader that defined it. And at any instant, JVM can contain only one instance of Class pertaining to a class/interface in the same classloader namespace. But it is possible that the same class/interface exists during runtime in different classloader namespaces. To avoid confusion and guarantee type safety (and security) in such scenarios, special validations called loading constraints are imposed so that a type even though loaded by multiple classloaders, refer to the same class/interface definition.
It is also worth mentioning here that arrays are handled differently from classes and interfaces – primitive array types are loaded by the bootstrap classloader and reference type arrays are loaded by JVM itself. The component type of the latter is loaded through classloader mechanism though.
3-Step Classloading Process
Classloading happens in three distinct steps – loading, linking and initialization. Loading involves building an implementation specific internal representation of a class/interface. This occurs either during runtime constant pool resolution (which in turn happens on method invocation, field access etc… of an already loaded class/interface) or during a reflective call. Linking makes sure that the loaded type (including its parent class and interfaces) can be safely used. And initialization as the last step sets the initial values of static fields. The specification mandates that even if there are errors during linking or initialization, they are to be thrown at a point in the program that uses the class/interface and strictly not otherwise.
The sequence of steps for classloading can be represented as:
- Load: This is the first step in creating type definition of a class/interface. Here binary information of the type is parsed; validated against standard .class format and stored in the method area and finally, a Class object is created. Direct superclass and superinterfaces are also recursively loaded and linked till Object class.
- Link: As shown in the figure, linking involves three sub-steps – verification, preparation and resolution.
- Verification consists of elaborate checks to ensure security and type safety of the bytecode. This includes structural validations and well-formedness checks of the .class file and trustworthiness checks of code for potential exploits by following well-defined type checking and type inference rules. Since this is a costly operation, .class files generated for JVMs of limited memory devices are often pre-verified by the compiler.
- Preparation involves creating static fields and initializing them to their default values.
- Resolution of symbolic references in the class or interface is an optional part of linking and is often done when the field or method is used for the first time. It basically involves determining concrete values of symbolic references in the runtime constant pool.
- Initialize: This involves execution of class/interface initialization method <clinit> of superclass and superinterfaces and finally that of the current class/interface. The <clinit> method is nothing but a synthetic method, equivalent to static initializing block, added by the compiler to set initial values of static fields.
Since JVM supports multithreading, it is possible that more than one thread may try to load the same class/interface at the same time. It is the responsibility of the implementation to guarantee that only one instance of class/interface is loaded within a classloader hierarchy. Often, the Class object which is created for each class/interface, in the first step, is used to synchronize the entire classloading process.
Cross-Context Communication between Web Applications
The Problem Scenario:
In one of our assignments, we faced a problem in making communication between two different spring based web applications which are in the same web container. Let us explain it with an example; say there are two web applications ‘Foo’ and ‘Bar’ deployed in the same web container and ‘Foo’ depends on ‘Bar’ to access a method say ‘barMethod()’ of ‘BarService’ ( a Java Class present inside the ‘Bar’ application), then
- The ‘Bar’ application servlet must provide a method to access its Java classes
- The ‘Foo’ application must get the response from Bar application through HTTP.
Problem#1: Though both the applications reside inside the same container and ‘Foo’ application knows what method of the Java class of ‘Bar’ application it has to access, it has to go through the network using HTTP protocol as if ‘Foo’ application request is considered as a client request to ‘Bar’ application as shown in the figure. Is there an optimal way to avoid the HTTP protocol overhead?
- Foo-Bar communication with out Adapter
Problem#2: Say a client (web, mobile etc..) invokes ‘fooMethod()’ of ‘FooService’ which internally calls ‘barMethod()’ of ‘BarService’ through ‘FooServlet’. And say the transaction involved in ‘Foo’ succeeds and the transactions involved in ‘Bar’ fails, then the first transaction gets committed and the second gets rolled-back as the participating threads are different. It means that according to the client, the participating transactions are not atomic as they get partially committed. Is there a way to make them atomic?
The Solution:
Yes..The solution what we found is to use an ‘adapter’ code in a cross-context enabled web container as shown in the figure, which would help in solving both the problems discussed.
- Foo-Bar communication With Adapter
-
Enabling cross-context:
Cross-Context is a feature provided by web containers and enabling configuration varies from container to container. For example, in Tomcat, the ‘crossContext’ attribute in ‘Context’ element of ‘TOMCAT_HOME\conf\context.xml’ must be set to true, to enable cross-context communication as shown below. By enabling this, one web application can get access to other web application.
________________________________________________________________ <Context crossContext="true">
<!-- Default set of monitored resources -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
</Context>
________________________________________________________________
-
The Adapter:
Once the ‘Bar’ context is obtainable in ‘Foo’, we can easily get the ‘Bar’ application spring beans using which we can access the ‘barMethod’. Yes!. That’s exactly is what the adapter does. Let us see how to invoke the ‘barMethod’ from ‘Foo’ context in two different sections. The first section explains the adapter functionality if the method invocation involves only simple data types (like int, float, String) as the method arguments/ return type and the second section explains the adapter functionality if the method invocation involves custom data types like (Person, Address etc…).
Handling Simple Data Types
The adapter does the following, before accessing any of the ‘Bar’ application Java classes (‘BarService’),
- Saves the current class loader (which loaded the ‘Foo’ application) in a temporary variable
- Sets the current class loader to the one which loaded the ‘Bar’ web application.
- Resets the current class loader with the one we saved in the temporary variable once it has finished accessing the Java Class of ‘Bar’ application.
- Let us see the adapter code.
___________________________________________________________________________________________
String methodName = request.getParameter(WEBConstants.PARAM_METHOD); ServletContext srcServletContext = request.getSession().getServletContext(); ServletContext targetServletContext = srcServletContext.getContext("/Bar"); //save the class loader which loaded the 'Foo' application in a variable ClassLoader currentClassLoader = Thread.currentThread(). getContextClassLoader(); try { Object object = targetServletContext.getAttribute ("org.springframework.web.servlet.FrameworkServlet.CONTEXT.bar"); // get the class loader which loaded the 'Bar' application ClassLoader targetServiceClassLoader = object.getClass().getClassLoader(); // and set it as the current context class loader. Thread.currentThread().setContextClassLoader(targetServiceClassLoader); Class<?> classBarService = (Class<?>) targetServiceClassLoader.loadClass ("com.pramati.crosscontext.service.BarService"); Method getBeanMethod = object.getClass().getMethod ("getBean", String.class); // Get the barService defined in the 'Bar' application context. Object barService = getBeanMethod.invoke(object, "barService"); // Get the method of the 'barService' Method targetMethod = classBarService.getMethod(methodName, (Class[]) null); if (targetMethod == null){ response.getWriter().println( "Error: The method['" + methodName + "' does not exist" ); return null; } // Invoke the method on 'barService' Object responseFromBarMethod = targetMethod.invoke(barService, (Object[]) null); } catch (Exception e) { e.printStackTrace(); } finally { Thread.currentThread().setContextClassLoader(currentClassLoader); }
_______________________________________________________________________
If we do not have these steps done, then we end up in getting ‘ClassCastException’ though we cast the right object with right type. Why? It is simple, the class loaders which loaded the ‘Foo’ application and ‘Bar’ application are different.
Handling Custom Data Types
Are there any changes if we need to pass custom type (Person/ Address etc..) parameters while invoking the method on the other context?. Absolutely Yes; we have changes to be made. There are two ways to fix this scenario.
Solution-1: If we could externalize the custom data type classes which are accessed by multiple web applications (here accessed by ‘Foo’ and ‘Bar’) to a different library and place it inside the commons library location of the web container (In case of tomcat, it is <TOMCAT_HOME>\lib), then we can pass the parameters as we did for simple data types without any source code modification.
Solution-2: If we can not externalize the common custom types then the custom data type objects which are going to be used as parameters must be serialized and deserialized say using JSON JavaScript Object Notation (or XML etc..). By including the below snippet to the adapter, we can do it.
___________________________________________________________________________________________
Person person = new Person(1, "Kayal"); ObjectMapper jacksonMapper = new ObjectMapper(); // serialize the param 'Person' with the 'Foo' application jacksonMapper String serializedPerson = jacksonMapper.writeValueAsString(person); Class<?> classPerson = (Class<?>) targetServiceClassLoader.loadClass ("com.pramati.crosscontext.model.Person"); // Get the jacksonMapper defined in the 'Bar' application context. Object targetJacksonMapper = getBeanMethod.invoke(object, "jacksonMapper"); //Get the 'readValue' method of the 'Bar' application's jacksonMapper Method readValueMethod = targetJacksonMapper.getClass().getMethod ("readValue", String.class, Class.class); //Deserialize Person using 'readValue()' of the 'Bar' application's jacksonMapper Object deserializedPerson = readValueMethod.invoke(targetJacksonMapper, serializedPerson, classPerson); // invoke 'barMethodWithParam' of 'BarService' with the deserialized Person responseFromBarMethod = targetMethod.invoke(barService, new Object[] { deserializedPerson}); // Get the 'writeValueAsString' method of the 'Bar' application's jacksonMapper Method writeValueAsStringMethod = targetJacksonMapper.getClass(). getMethod("writeValueAsString", Object.class); //serialize response using 'writeValueAsString' of 'Bar' appln's jacksonMapper responseFromBarMethod = (String) writeValueAsStringMethod.invoke (targetJacksonMapper, responseFromBarMethod); ___________________________________________________________________________________________
With this,
- Able to improve performance by avoiding the HTTP protocol overhead.
- Able to achieve atomicity in transactions.
How? The transactions are cached by the Spring Tx. Manager using ‘Thread Local’ concept which are specific to threads. If we have communication between applications using the adapter, the threads and the participating transactions are same; hence is atomic.
Sending messages to nil references in objective C
This might come as a surprise to most programmers, even experienced programmers – in Objective C, messages can be sent safely to nil references.
NSString *name = nil;
NSInteger length = [name length] // length = 0. Surprised?
This is perfectly legal. Another surprise, length has a valid value, zero, and not any garbage value.
Hope your interest is perked up. Let us first set the context for this discussion by comparing other languages.
Behaviour in C is pretty well known, SEGMENTATION FAULT which invariably crashes the program unless the SEGSEV signal is handled. Even if its handled there is nothing much that could be done because there is no way to go back to the previous point of execution.
Now Java tames this ugly beast. An exception is raised and if it is handled properly, the application can even continue execution. But the code becomes ugly with so many try-catch blocks littered all over even while dealing with simple operations.
Objective C completely tames the beast, it not only not crashes, but the programmer could be almost assured that the return value might be valid. This seems a bit vague but there are rules which govern the value returned by messages sent to nil objects.
- If the method returns an object, any pointer type, any integer scalar of size less than or equal to sizeof(void*), a float, a double, a long double, or a long long, then a message sent to nil returns 0.
- If the method returns a struct, as defined by the Mac OS X ABI Function Call Guide to be returned in registers, then a message sent to nil returns 0.0 for every field in the data structure. Other struct data types will not be filled with zeros.
- If the method returns anything other than the aforementioned value types the return value of a message sent to nil is undefined.
In short, return values from messages to nil objects are guaranteed to return the equivalent of zero for scalar return types, i.e., nil, 0, 0.0, NO, etc.
One disadvantage of this behavior is that bugs may go unnoticed. Objects which should not be nil might be nil but the code might behave as if nothing is wrong and it is tough to catch bugs in early stages of development.
But beware, sending messages to nil should not be confused with sending messages to deallocated instances.
[obj release] //assume retain count becomes 0 and object is released.
[obj doSomething]
The above code releases object but does not nil the reference variable. Any messages sent to this reference variable will invariably lead to a crash.
This proper way of releasing an object is
[obj release]; // assume retain count becomes 0 and object gets released
obj=nil ;
[obj doSomething] // perfectly safe
The same effect maybe achieved using
self.obj = nil // assuming the property is declared with retain property
But this might cause KVO invocations which might not be needed.
All in all, the main advantage (maybe the only) advantage with this behavior is that code gets simplified and becomes more intuitive. Unnecessary checks which get littered all around can be avoided keeping the code more readable. In case an explicit check is required, the programmer can go ahead and add it anyway. So all in all, I think objective C has a good way of handling nil objects and maybe other languages could adopt it.
Any thoughts?
QuotientCube
For one of our new initiatives we were looking at quotient cube. Taking the liberty of few over simplifications we can look at quotient cube as a precomputed cube for all the dimensional combinations available in the data which preserves the navigation(rollup/drilldown) semantics and is very concise. Using a tree representation for the same helps us keep the retrieval of any dimensional combination aggregation to k where k is the number of dimensions.
A quotient cube takes care of dimensional sparseness and also helps us in understanding which values may not have a say in a particular scenario. Lets take a simple case of 20 yr sales data for apple products. Time as usual would be one of the dimensions in it, computing the aggregated sales value for iPad for years before 2010 does not make sense, as iPad was launched in 2010. Since quotient cube computes and stores aggregates only for value combinations available in the data, it would not waste any computation on calculation of sales figures for iPad before 2010.
The data structure and the values that are stored facilitate programmatic discovery of the fact that iPad sales are not applicable for these years and could report this as an interesting trend.
We are trying to come up with a js implementation of the cube which can be run in the browser or using nodejs. Thanks to the awesome performance of modern day js engines, we should have a simple js include to convert any html table to a data cube. Our initial experiments with movie lens data prove that we can easily crunch about 100k records at server side in 2 seconds on an average laptop using node js, for a pure client side implementation without workers, 1.2k records can be processed without breaking any sweat. The other implementation is as an UDF for MySQL.