JDK/PRT (JPRT) Build/Test System Status
Status updates will be sent to jprt-announce@sun.com, a NetAdmin alias anyone is
free to join.
Relationship to existing PRT source
The original Hotspot PRT manual is here:
http://j2se.sfbay/web/bin/view/LabEast/PRTManual
and the original Hotspot PRT source tree is at /net/prt-web.sfbay/prt/prt2-src.
This new JDK/PRT (JPRT) master source tree
will be kept at /java/svc/ws/jprt.
Initially I assumed I could just make "less than major" changes to the existing
PRT system and add in the ability to build the j2se or JDK source tree.
Early on I abandoned that plan and although I am re-using a great deal of the
existing PRT system, I have also made extensive changes to what I have used.
Much of this is cosmetic, by simply cleaning up code, adding javadoc comments,
etc., but to say it is only cosmetic would be misleading, the changes are
pretty extensive and beyond the abilities of filemerge to deal with.
Some major problems with PRT have been fixed, e.g. the ByteBuffer/nio problems
where the files being transferred would cause OOM's.
At first the code merging was acceptable with other PRT changes, but as I
started using NetBeans more and more, and I started cleanup up parts of the
PRT sources that needed cleanup or simplification, the merging became very
time-consuming and de-stabilizing. It just didn't seem worth it to me.
Originally the PRT designers had a prt '-jdk' option in mind, it was just
never implemented. Unfortunately, this unimplemented '-jdk' option would have
negated all the cool hotspot specific building/testing features of the
PRT system that I wanted to have in a JDK PRT system.
So the '-jdk' option was abandoned in favor of re-working all the existing
hotspot specific code in the PRT to be more generic.
The ability to select the build targets and the test targets
is configured with a make/jprt.properties file in the source tree.
The JPRT build process assumes there is a set of 'jprt_build_product',
'jprt_build_fastdebug', and 'jprt_build_debug' targets in the
make/Makefile.
This allows for the source tree to customize what is built, and also
what is bundled.
Each of these targets results in a single bundle or zip file that is what
is saved on the driver side.
The use of NetBeans 4.1 (and then 5.0 and 5.5) was mostly considered
to be an educational experience,
but as I used it more and more, it became fairly essential to working on the
code. I became 'addicted' so to speak.
Inside NetBeans it is easy to re-structure, re-name, format code, fix import
statements, encapsulate fields, add javadoc comments, edit javadoc comments,
create JUnit tests, etc.
This resulted in such massive changes and file movements and file re-names that
syncing this up with another source tree was difficult if not impossible.
Making it worse, it seems like filemerge isn't horribly helpful with merging
Java code, it doesn't know any language syntax, and even if it did, this could
be pretty difficult.
So I gave up on a formal merged source tree with PRT for the time being
and am just working with NetBeans and a raw
source tree, mostly on my Mac PowerBook.
Specifically the source has changed in these ways:
-
The path to the product itself is now automatically discovered via the
classpath, so a PRT_HOME like environment variable is no longer needed.
-
Build targets can be specific to an os release, e.g. solaris_sparc_5.9.
Names without the trailing OS rev number will assume to mean the oldest
release for that osname and arch.
-
Solaris sparcv9 and x64 clients will be running their jprt instance
with -d64, and need to in order to automatically identify it as that 64
bit arch. These are treated as separate platforms.
-
The property files are now part of the source tree and are
included in the classes tree and the jar file "jprt.jar" for easy locating at run-time.
The primary property filename is "JPRT/jprt.properties", and the system will
look for a jprt.domainname.properties file for domain specific settings.
The domainname properties file has the bulk of the configuration settings
needed to create a new instance of a jprt system.
The machine configuration information is in the sbin/jprt_config* files.
-
Re-named the product JPRT and the tools start with 'jprt_' instead of 'prt_'.
All property and environment variables start with JPRT.
-
The job continue action is now a separate tool, jprt_continue jobID, rather
that having the prt_submit command have a -continue jobID option.
-
All the tools have consistent argument parsing rules, and all have a
'-usage' and '-help' option.
The option spelling and meanings are shared and consistent between tools
(as best as possible).
-
Added extensive javadoc comments throughout.
Copied in many of the comments from the PRT manual on the commands into the
help messages for the commands, and into the javadoc comments.
-
Done quite a bit of re-packaging (renaming packages, moving classes around).
-
Folded together duplicate functionality (e.g. ShellCommand class was removed and
replaced with the ExternalCmd).
-
Created many shared methods
(replaced some often repeated 6-10 lines of code with a ExternalCmd method call),
-
Created a PlatformID class (for any machine, not just build machines,
and with a self identifying feature too).
-
Created a JdkReleaseID class (that will return a list of build PlatformIDs and test PlatformIDs).
-
Isolated the BuildTargetID and TestTargetID string ID manipulations.
Also removed the compiler name from the build target.
-
Isolated the use of System.{exit,out,err},
-
Isolated all access to the property values to Globals.
-
Deleted many unused methods in many classes.
-
Changed many methods and fields to 'private' when they were only used within the class.
-
Did lots of code re-formating (with NetBeans and NetBeans default rules).
-
Expanded all import statements to be explicit one per class (no more import wild cards).
(The wild cards were a problem with many classes due to class name conflicts, and in
general I like to see each and every class imported, makes the dependencies clearer).
NetBeans has a 'Fix Imports' option that keeps the import lists cleaned up.
-
Tried to de-PRT and de-hotspot and de-vm the code in terms of those names.
Also the name "prtd" was replaced with "driver", the vm clients are just
the clients, and the term PRT or prt is just the 'system'.
-
Localized all constant strings to Globals class (default machine names, paths, email
addresses, etc.).
-
Reduced, re-named, re-structured the properties and allowed for common property keys
among common OS's (e.g. a solaris
property for all of solaris, but overridden by a solaris-sparc property if necessary).
All the property names start with JPRT now, and some cleanup and simplification
of some of the property names has taken place.
-
Folded together the two utility classes into a single shared one, MiscUtils.
-
Created a single 'main' class for the clients.
-
Made sure all uses of arrays and lists were generic'd.
-
Replaced the use of the Iterator class in many cases with simple for loops (JDK 5.0 specific).
-
Got rid of all warning messages, including those with 'javac -Xlint:all' (except for
some generic issues with the Element class and the jdom.jar file).
-
Removed warnings found by running nthe PSD Netbeans module (style checker mostly).
-
Fixed a few bugs found with findbugs, but findbugs is way too noisy for me.
-
And more changes that I have probably forgotten.
I have not paid much attention to the Legacy issue of parsing old build and test
target id's.
The biggest legacy issue will be dealing with the legacy file layouts, which will
be changed pretty severely in this new JPRT system.
What remains to do
So what is left?
-
IN PROGRESS: Test: conflict jobs, jprt_continue, jprt_extract, jprt_setacceptingwork,
jprt_stats, coredumps, kills on windows, more testing on windows
especially on kill jobs.
-
IN PROGRESS:
Deal with import and bootdir jdk issues. Be able to select, and have the jdks
automatically installed on clients as needed, or when requested?
-
Investigate what it would take to build jdk5? Or jdk1.4.2? Should just need
to construct the correct make/jprt.config file for each source tree.
Investigation donw. Need jprt.config, jprt.properties, and Makefile rules
in jdk5 source trees.
Client machines need jdk5 components, e.g. compilers etc.
Windows i586 machines will need 6.2 installed on them.
Windows x64 should be fine as soon as the jdk5 builds switch over.
Solaris: jprt_setupclient was changed to also install SOS8 and SS10 for jdk5.
Linux: should be the same.
-
DONE:
Figure out how to get 'ssh -l administrator' to work on windows machines
so I can use jprt_start/jprt_stop.
-
DONE:
Use rsync to only bringover the parts of /java on a client machine that are
needed.
-
Verify the JDK builds are NOT accessing /java or J:/ files.
-
DONE:
Deal with the environment variable settings per sourcetree/platform/release.
Add a jprt.config file to each source tree?
Also deal with what environment settings get passed to ProcessBuilder.start().
-
Figure out how to run the existing junit tests from the Makefile on
Solaris.
-
IN PROGRESS:
Define and implement the "source base set" and "primary source base" concept.
I'll need this for control builds where more than one source tree is involved.
More thought needs to go into this, it's a nasty issue of doing putbacks, but
may be workable as a build/test only, no integrations.
Created a SourceFiles class to hold all info on where sources come from.
Currently has a source bundle mechanism.
-
Deal with the legacy job, build target, and test target ID issues.
Not sure how important this is.
-
Deal with the jdom.jar dependency.
The access to these jdom classes has been isolated and an attempt was made
to change to the classes in the JDK, but they are different enough that I could
not get this conversion to work.
Both implementations of XmlElement are in the source base, we just need to
get the JDK one working and delete uses of the jdom.jar classes and jdom.jar.
-
Add junit tests for the more critical and well isolated classes once they
have stabilized and are no longer changing, if possible.
This is an on-going task.
-
Send logging messages from client to driver? Keep all logs on driver in one file??
-
Fix bringover performance issue. The j2se source tree is too big and 20-40 minutes
is taken just getting a source tree in place.
Mirror source trees (tricky with undos?) would work, but need to be
automatic.
PRT mirror source trees were not automatic.
-
IN PROGRESS:
Fix testing issues so that we can add j2se tests.
-
IN PROGRESS:
Add j2se tests!
-
IN PROGRESS:
Work on DiskCleaner and have it stage finished jobs to the storage array archive
automatically.
Get storage array up.
-
IN PROGRESS: (experiments on east coast)
Try transitioning PRT system to JPRT (Keith/Steve trying on east coast).
-
Order more hardware.
Hardware Status
The initial hardware arrived in November 2005.
The rack arrived several months later.
The rackmount rails had to be ordered special and caused more delays.
The hardware is finally being installed now (April 2006).
The initial JPRT system with this hardware was up and running in May 2006.