|
On my way back from San Jose California, I was taking a beer break waiting for my flight back to Orange County when I met Mark. He introduced himself as Chief Architect for a fortune 100 company. Along with our casual airport chat, we also discussed Global Single Instance strategy. Mark was architecting his company’s GSI (Global Single Instance) consolidation project. It was the second time in less than 1 month that I met somebody who was talking about GSI. And ironically, I was leading a GSI project in the Bay Area. It was my fifth GSI architecture project in nine months where I have either presented solution architecture papers to the executive team, implemented, them or both. One common thing was visible in all occasions. All of the clients were globally located with multi-channel business and organizational structures. Producing a consolidated financials or inventory reports was cumbersome. The IT was spending much for integrating, customizing and maintaining their diverse infrastructures. Costs for license and software maintenance were sky rocketing. All these factors added to be an excellent case for having a GSI for the business to advance ahead, cut cost and be on the competitive edge. Are you ready? How do you balance business with the cost of maintaining a globally disperse systems? What is the best approach for a single instance consolidation? How costly and lengthy would it will be? Should I utilize my own staff or invite a professional consulting firm? What implementation model I should follow? The questions are many. And, the answers are many, and it all depends on a number of factors of your business. But in a nut shell, architecting a Global Single Instance is indeed a lengthy and costly investment, but the ROI is rewarding. The idea of Global Single Instance strategy is to consolidate to have a single source of truth available globally for all financial, inventories and other real-time reporting. Especially, when large corporations need to follow strict financial regulations The following article emphasis an Oracle e-Business suite using shared APPL_TOP technology utilizing RAC to consolidate globally diverse systems. The objective of this article is to explain a general idea to build a foundational architecture framework so that you could expand it with any addition of processes or technologies without much impact to the operational processes. Further, it expands the basic architecture enabling scalability and high availability by leveraging Cluster and Load Balancing technologies. Now, Lets us take a quick look into architecting and building a Global Single Instance and the important process areas that need to be considered. Knowing Your Business Understanding your end-to-end business processes is sometimes a big task. And in most cases, not everyone in the company knows what they are doing. The task should include deliverables of defining an organizational chart, its objectives, its responsibilities, location, inventories, products, people and technology. A series of this exercise for specific location would help business to define an entire organizational structure along with their boundary system definition that could sometimes stretch globally. Once you have this boundary system defined you get an over-all picture about possible integration points, security path, access path, external and internal users, the diversity of data across the globe, typical latency of regions etc. Service Level Agreements Whether it is with the business or vendor, this is one of the key parameter for deciding the framework. Definition should include a ‘point in time’ performance and load characteristics of production servers, the acceptable slowdown or availability, acceptable response time of the applications etc. This point in time availability may sometime be the main driving point in choosing the server configurations. Typically business would require 99.99% availability, but for some case this may be ‘Four nines’ or ‘Five Nines’. End Users Identifying end users, both external and internal is another important area. The deliverable should include the user, role, privilege, applications and access control, type of user, location and time zones and hours of business. This will provide greater details about the load characteristics, performance characteristics, latency requirements etc. Volumetric Volumetric is a nice to have thing and may not always accurate. But the general idea is to get an estimate of the consolidated size of the existing data and estimating the data retention period and projecting a possible future size of the data. A confidence factor is lined against each data source in order to get the variation of the actual data. A 20-30% overhead depending on the confidence factor may be added to estimate a rough ball park. Building the basic framework Once you have this information, you could now start drawing the foundational architecture framework. The framework should be simple and yet expandable on demand. It should also contain place holder for all components and should define an easy integration layer between the components without affecting each other. Security and access control mechanism should be also included for both external and internal users. Since external users can also be an internal user, special exception rule should be included if an internal user accesses the framework externally. It should also include uniform access control and central monitoring, managing and reporting option. Finally, the framework should be able to unplug a single component and redesign them without much complexity or interruption. Let us try to explain these concepts by using the following logical figure. 
DMZ Layer This is the network area that separate between the trusted intranet and the outside connections. The main objective for this is to let external user access the network. The access method can be DNS, HTTP, HTTPS or VPN. Other protocol may be allowed depending on the security requirement. In this layer the foundation can utilize external facing web servers, application web servers, single sign-on web servers, public load balancer etc. Ideally, for security reasons, this layer should allow only HTTPS connections on port 443 inside and all other ports should be blocked. Load Balancer Layer Load balancing may become highly necessary in a global single instance. Mostly this is because the high number of users accessing the servers, the type of users (external or internal) and the function, whether the user is looking for a status of an order that has placed or if the user is accessing the inventory as super user.The load balancer will take the external request and balance with the available application web server components within the trusted intranet. The load balancer could be hardware or software based. If using software based load balancer, then additional server to host the software and network configuration is needed. For hardware load balancer this does not apply. An optional firewall is placed to further filter the ports and control the access between intranet and the load balancer. All connections from external network are channeled only through the load balancer. Commonly used load balancer products are Cisco Network, Foundry Network, Coyote Point Network, Citrix Networks, Barracuda Networks, Radware etc. Intranet Layer The intranet layer is the trusted network for the application. A corporate intranet may be further segmented into network zones according to security, traffic, accessibility and other factors. In a GSI environment, to secure direct access of any application layer it is recommended to have a network zone and only authorize load balancer and certain required components to access the application layer directly. This will prevent un-authorized access request directly to the application and will enhance the single point of monitoring and access control mechanism. This may be absolutely needed for businesses that deal with Federal customers. Web Layer The application or the web layer provides intermediate access to the data, manage processes, and integrate processes or monitor of all activities. The basic type of application on this layer is the web services. These services can use HTTP, HTTPS, XML, SOAP, and WML to intercept, manipulate, connect and integrate with the requests coming from the intranet or load balancer and pass it on to the subsequent layers. The ease of this framework allows any component that runs on a web service to be plugged in or out without affecting other applications. Some of the applications that utilized this layer are application web servers, authentication server components for identity management, services that utilizes MOM or J2EE based queues, integration servers that include data transformation or data loading. Forms Layer Forms layer uses strictly TCP connections utilizing SQL*NET. This layer connects directly to the database layer. A separate layer of firewall can be introduced and the forms port and web ports are filtered for access. Doing so will provide additional security to the database layer. This layer may include the concurrent managers which also uses SQL*NET base TCP connections to the database layer. Database Layer The database layer is where the core RDBMS engine runs. Certain web servers, forms and concurrent manager servers are only allowed to have connections to this layer. Further more, these are filtered by ports and protocols to minimize unauthorized access to the database. Storage Layer All physical data files are stored in this layer. Most commonly used file systems are SAN, NAS and some case NTFS, HDD etc. A dedicated access path should be defined between the database servers and the storage layer to achieve best I/O throughput. These connections should be private FC networks with at least 1G bandwidth. Depending upon the server NIC availability, the network could be trunk to boost the bandwidth later. Sometimes this network can also include the tape devices for backup and recovery. Enabling High Availability To achieve four or five nines of system availability, sometimes huge investments may be needed upfront. But this will pay off at a long term. As discussed earlier, the SLA would define how expensive and robust your infrastructure should be. Depending upon the applications running, there are options of having a number of mid-range servers clustered to form a scalable and high available farm, or a small number of high-end servers with massive computing power to form the cluster. But, having a cluster does not guarantee the four nines availability. To achieve a fully scalable and high available system we need to secure HA and scalability for the following components. • Load Balancers • Application Web Servers • Forms Servers • Identity Management Servers • Database Servers • Storage • Networks Further, we need 2n or 4n symmetric replications of the infrastructure. These symmetric clones will provide HA (High Availability) or MAA (Maximum Availability Architecture) depending upon the business needs. HA for Load Balancer Load balancer is the primary entry point to the application. To establish availability to this entry point, a symmetric environment should be created. The architecture should be able to provide availability not only the primary site, but also to the secondary site to fail over. The following figure explains the general concept of the configuration. 
The external entry point from the internet is filtered by the firewall to allow only HTTPS or HTTP depending upon the application. The symmetric firewall configuration provides availability in case one fails. The load balancer will now load the request to the available web server clusters thus providing not only load balancing capability, which increases performance, but also uniform distribution of users. Further, with certain extra configuration on network and web servers, users could be routed to specific set of web servers based on user roles, IP filtering, DNS filtering or user responsibility etc. HA is established by using load balancer. At any point of time web servers can be easily added for scalability. For HA testing, keeping W1 through W4 down, an even load balancing is observed for W5 and W6. When W5 was taken down with users connected, the application URL has to be refreshed to establish connection using sustaining W6. But normal behavior was observed for new user connections. (ie. New user acquired connection as usual). Thus for external users who are not internal users, could now route to web servers W1, W2 and W3 and internal users could route to W4 and W5, W6…etc. A similar infrastructure is maintained on secondary site to achieve MAA. The primary and secondary site could mask a common VIP for the public access allowing users a transparent fail over to either sites. The method above provided high availability for the load balancing services, performance and scalability for web servers. HA and scalability for Web Servers At load balancer layer, three virtual address are created using VIP’s. These addresses are used for (1) first point of entry from the internet zone, (2) the single sign-on authentication web server and (3) the application web server. When installing eBusiness suite first time, the application web URL should be provided as the first entry point to enable the load balancing (ie. http://www.entrypoint.com). This address should now pass the token to the single sign-on server, which is another virtual address created at load balancer layer (ie. http://singlesignon.com). The single sign-on will set cookies and pass the authentication to the application web server, which further passes the token to the forms server to enable a direct connection with the database. Each web servers can act independent and should be configured using a common web URL (ie http://webserver.com). Further nodes can be added using autoconfig tool and should be given the same URL. Load balancer provides cookie stickiness and persistent function that will allow load balancing web servers correctly for same user request session. The default random method can be chose for load balancing. Enabling HA for Forms Servers Traditionally, forms server used metric load balancing technique. The master node co-ordinates the load activities among the child nodes and evenly balance the load according to the statistics returned from the child nodes. The drawback of the above technique was that if the master node goes down, all child nodes are also effected until a new node is manually (or scripted) re-configured. This unplanned downtime may sometimes range from 10 minutes to 1 hour depending upon the environment. In another method, avoiding the metric load balancing, the web server and forms servers are coupled together in the same box and balanced through a hardware load balancer. This new web/forms server used JSERV engines to load among the forms and used session cookies and stickiness set at the web configuration files. Again, the draw back for the above configuration was either a forms or web issues could bring the entire failure. Also, with the JSERV engines, performance tuning and maintenance efforts were high. This will affect the overall availability and the SLA. A different approach can be utilized to introduce HA for forms servers. In this architecture additional set of load balancer is introduced between web servers and the forms servers. You could use the same set of load balancers, but to isolate the internal traffic to outside, it is good to have additional sets of load balancers. A VIP is created for this layer (http://formsserver.com) for pointing to a single forms server for load balancing. All requests coming through the web server now redirects to this VIP for load balancing. These requests are now routed evenly to the participant nodes. These nodes can act independently and behaves to the application as a single node configuration, yet providing fully scalability and high availability to the forms server nodes. While configuring the forms server the application web server URL is referred as dummy http://webserver.com. On the other hand the web server is configured by giving the forms server URL as http://formserver.com as the forms entry point. The example is given in figure below. 
Advantages The advantage of this architecture is that we now isolate forms server with the web servers. Any single node failure on either side will not affect both nodes function. As we introduce a flexible foundational framework, the web layer is the common integration point to all other applications and services in a SOA based environment. These services should be separated and strictly isolated to reach the forms layer directly. This is very important from a security stand point, since most of the times the connections between the forms and database layers are open TCP using SQL*NET. Scalability of forms server is now easy to achieve. By using autoconfig tool, further forms node can be added easily without affecting other activities by using the common forms VIP address. HA for Identity Management Servers With the certification of Oracle Application servers 10g with e-Business suite, we can now centrally manage the authentication and access control through a single repository under Oracle Internet Directory. In coming versions, this feature would be an integral part of e-Business suite 12i and the fusion applications beyond that, which will replace the current architecture of 1.0.2.2.2 application server and introduce the Oracle applications 10g framework. Without enabling HA for these components the availability of the entire framework may be in danger. For the scope of this article, we will discuss a highly secured environment where three-tier architecture is used. The first layer is the authentication web server or the OHS layer, the second tier is the applications layer that runs the Oracle Access Manager, the third layer is the Oracle Internet Directory. We will discuss HA for these components as below. Enabling HA for Infrastructure (OID) The OID repository is stored under an Oracle RDBMS. The current certification allows leveraging 10g R2 RDBMS to store this repository. Thus, it becomes essential to have this repository data available at all time. The Oracle Real Application Cluster provides not only high availability, but also the performance and scalability and the ability to integrate with a single Oracle Enterprise Manager Grid Control monitoring console. Let us discuss further using the following figure. 
The first step is to create an Oracle Real Application Cluster database. Now, set up the first infrastructure host and have the repository created in this RAC database. When naming the infrastructure farm, provide the VIP that we have created on the load balancer (ie http://oidhost.com). Load balancer is configured to filter requests from both infrastructure servers. The second infrastructure host is now added to share the existing repository using the same OID URL. Once complete the OID will have a common URL that will load balance among two infrastructure hosts and provides HA to the database using RAC. A firewall is placed to isolate the OID hosts with the middle-tiers servers preventing direct connection and filters only the relevant secure ports. Most load balancers would allow filtering secure ports 636, 443, 389 and should allow HTTPS protocols. In some products, these ports may be not available and could stop the installation of the second host. If that fails you could create an alias on Oracle RAC VIP on the host files to point to http://oidhost.com to complete installation. HA for Identity middle-tier server The middle tier layer consists of Oracle Access Manager, Identity and Access servers, LDAP and the SSO Servers that could be deployed as tier-one or tier-two architecture. To follow the foundational architecture, a two-tier layout should be followed as below. 
The goal is to separate the web tier with the LDAP tier. A VIP is created in the load balancer to balance the middle-tier hosts (ie http://middletier.com) The Identity management hosts are then installed without selecting the SSO. The infrastructure host URL given as http://oidhost.com VIP and the web host would be given as http://singlesignon.com VIP. After installing the middle-tier, further installation is done to setup the web tier SSO component. The common web host is again would be http://singlesignon.com that will be the entry point to the Identity Management Component. The Oracle e-Business will now pass the token from http://webserver.com to http://singlesignon.com which will pass the token through the middle-tier LDAP tier http://midtier.com to the OID repository http://oidhost.com for authentication. Once authenticated, the cookie is set and the connection is passed back to the application web servers to allow users to login. HA for database servers Enabling high availability for the database servers are critical for not only for e-Business applications, but also for other business applications. One of the best technologies available to achieve this is using Grid technology. The concept of having a pool of available servers to work as one, on demand, that is transparent to location and users. Oracle’s Grid computing technology has been improving faster and steadily since its first concept of Parallel Servers. The new Grid uses the cache fusion technology that will make the transfer of data between the nodes in a cluster faster and efficient. Oracle Enterprise Grid Control console will enable to centralize all monitoring and maintenance routines reducing cost and DBA time spend on trouble shooting issues. Oracle e-Business suite has been certified to work with 10g R2 databases. Although, the package still comes with 9.2.0.6 database engine, an upgrade path to 10g R2 is easy. In future versions, the 10gR2 engine will be an integral part of the package. The use of rapid install wizard helps to implement the RAC easier than before. HA for Storage and Network To complete the robust framework, a redundant symmetric storage and network architecture need to be built. This should be for both primary and secondary sites to enable fail over and availability. The final framework should follow the best practices for performance as well as stability and maintenance. Securing your environment Securing your framework is the most complicated task. Since many of the components integrate with the framework at different points with different ports and protocols understanding the functionality of each application is very important. Defining security profiles and responsibility should be carefully planned with the implementation. To tightly secure the environment, each layer could be contained within different network zones with filtering within zones or domains for specific applications or ports. Thus, the database could be in “zone A” filtering only the forms layer that is on “zone B” to connect. Similar all other components can be zoned. With new features, the access can be restricted with user roles and responsibility in the application. With coming versions, multiple application web server URL can be defined for e-Business suite, allowing separating external and internal user login process. We could then use zoned SSO web servers and can load balanced using domain filtering to separate the internal and external user authentication. If needed a separated internet directory can be maintained for external users and could federate with the master directory. This can couple with any existing LDAP or Identity management directories to have an enterprise wide portal for single sign-on. The use of CoreID and Federated Identity management will allow consolidating multiple directories into a single source. Along with this, strict password policies should be set. Unsuccessful password retry entries should be logged. Replicating Secondary Site. Although primary site provides scalability and high availability, it does not guarantee against any disaster recovery. To fully maintain MAA, the primary site should need to be replicated to the secondary site in all respect. Ideally, the secondary site should be an exact replica of primary. But, for practical purpose this can be differ with the business. One of the major concerns is to sync data between primary and secondary site. Oracle Data Guard provides an excellent method to monitor, propagate and maintain a transactional consistent image of the primary. The Data Guard broker coordinates and maintains automatically all Data Guard configurations. On the secondary site Oracle Data Guard can create the copy either by SQL apply (Logical Standby) or redo apply (Physical Standby). Oracle e-Business is supported only to use redo apply method. Once created, Data Guard will automatically transport and apply these redo logs at the secondary site. Depending upon the SLA defined and the network structure Data Guard can choose to work in Maximum Protection, Maximum Availability or Maximum Performance mode. The distance between the primary and secondary site and the available network bandwidth along with the volumetric of business transactions per hours and the redo size could use to gauge the redo transfer time between sites, which can be use to determine which Data Guard method to choose. Reporting Architecture To prevent long running jobs that would potentially degrade the system performance, all reporting requests should be routed to dedicated servers. At least two nodes should be defined as parallel concurrent processing servers. One of these nodes could act as a master node that is powered up with processing capabilities. The secondary node or the fail over node can be any of the available nodes in the RAC, or it can be distributed among all nodes in case the master node dies. This can be configured by simple service names. This can be done by two ways. Method one can be explained with the help of the following figure. Considering an SLA defined to have 30 CPU’s available at any point in time. 
Node4 is dedicated for all concurrent processing for all normal operations. The node uses 30 CPU (illustration only). The node is identified by a service name PRIMARY. The secondary node is distributed across the remaining three nodes with a service name SECONDARY which have 40 CPU’s each. In case of primary failure, the secondary concurrent manager kicks in with the service name SECONDARY. The process would still have a 10 CPU ratio available to have a total 30 CPU processing power that would keep the SLA of 30 CPU. Second method is to have only a single node dedicated to be the secondary. Thus Node1 could now have a maximum of 70 CPU counts that can be dynamically allocated for taking the concurrent processing load in case primary dies. 
Other approach is to separate completely certain reporting those are long running into a separate environment. This separate environment could be built on separate physical servers to avoid any performance problems. Certain SAN solutions allows to copy from a snapshot which is easy to replicate the reporting environment. Thus, any jobs that requires more than 30 days worth of history should be run from this separate reporting environment. To create such, the programs need to be given privileges and permissions based on roles and responsibilities. Maintenance To build a comprehensive automatic monitoring mechanism, Oracle Enterprise Grid Control could be used. The framework will easily integrate with the foundational architecture and could easily be coupled with all components through management packs. The tool allows you to monitor not only Oracle components, but also the network, the load balancer and other tools that could be integrated with the SDK. OEM allows us to manage the cluster, add nodes, delete nodes and various backup jobs that can be programmed and monitored. It has also ability to automatic performance tuning and provides trend analysis of space and growth etc. Enterprise manager can also integrate with the application server framework with additional management packs to have a consolidated view of your entire enterprise application framework. Conclusion Planning and implementing a single global instance starts with strategy. The strategy should include evaluating the current business processes with the built in functionality available in the application, the changes and the impact of that to the business processes, people and the technology. Along with this, the architecture framework evaluation should leverage using a consolidated plug and play foundational architecture using Grid computing and SOA based components. This will easily allows changing the environment with the agile business needs. Finally secure your environment utilizing a single sign-on portal architecture that could consolidate and pool all lines of business along with leveraging an enterprise wide tool for monitoring and maintaining your entire framework. Among the other factors to consider, network latency and segmentation plays a key role in the design. This may become a bottle neck for certain application especially when users are spread across globally. Some of the Asian regions would not be a suitable candidate for having traditional forms layer due to the restriction of available bandwidth. About the author: Sam Chakkanat has been in management consulting industry since 1992. Prior to founding CEAUG, he was the Vice President of Enterprise Architecture practice at Agilex Technologies, servicing fortune 500 clients. Before that he was Enterprise Solution Architect and consulting manager at Oracle Consulting at their Platform and Performance Solution Architecture Group and had architect and managed multi-million dollar programs and business technology initiatives for clients. Mr.Chakkanat holds a bachelor's degree in electronics and communication engineering from Pune University. He serves as a though leader and subject matter expertise in multiple domains in IT strategy and Enterprise Architecture. He can be reached through our contact forum. Copyright © 2006 CBENCHUSA.COM . |
