Reading and writing fixed length records with Castor XML

I’ve been searching a generic Java tool in order to read and write fixed length records files with no success.

Traditionally, mainframe systems produce plain text files filled with fixed length records. Number records are filled with zeroes at the left (p.e. the fixed Number5 value 100 should be 00100) and String records are completed with blank spaces (p.e. the fixed String 5 value "abc" shoud be "abc "). So, this files may have the following aspect:

010011 CDOGenResCDW27032009
05001RES ACP 1 1 127032009
06001000000001Cosa1
06001000000002Cosa1
05999000000000004000000000002
05001RES REX 1 1 127032009
05999000000000002000000000000
01999000002000000000008

Read and write operations in Java over this type of plain files use to be tailored sentences using Java beans and formatter transformations. This technique can be a fast way to develop, but in my opinion decreases maintenance capabilities drastically.

Below a method to manage with plain files filled with fixed length records.

1. Develop an XSD to describe file contents



  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
    
        
       
    
  
  
    
        
    
  
  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
      
          
      
  
  
      
          
          
              
                  
                      
                      
                      
                      
                  
              
          
          
              
                  
                
                    
                        
                            
                            
                            
                            
                            
                            
                            
                        
                    
                
                
                    
                        
                            
                            
                            
                        
                    
                
                
                    
                        
                            
                            
                            
                        
                    
                
                  
              
          
          
              
                  
                      
                      
                      
                  
              
          
        
      
  

Note. “xs:” replaced by “xs_” for visualization purpouses.

2. Use some marshalling tool to generate Java beans (Castor 1.1.1 for this example)

3. Write some useful read and write tools (only “interfaces” included in this post)

FileWriter.java

public interface FileWriter {
  public FileWriter(String path, String fileName) throws IOException;
  public void writeNext(String line) throws IOException;
  public void endWrite();
}

FileReader.java

public interface FileReader {
    public FileReader(String path, String fileName);
    // Read one line
    public String readNext() throws IOException;
    public void gotoPosition(long position) throws IOException;
    public void endRead() throws IOException;
    public long getPreviousPosition();
}

LineWriter.java

public interface LineWriter {
  public void addNumber(Object number, int length) throws Exception;
  public void addString(Object string, int length) throws Exception;
  public String getLine();
}

LineReader.java

public interface LineReader {
  public LineReader(String line);
  public String getString(int length);
  public void reset();
  public boolean isEnded();
  public String getCurrentLine();
  public int getCurrentPosition();
}

4. Core class to write and read plain files using Castor objects

/**
 * Read and write plain text files filled with fixed width records using
 * Castor objects that represent XSD files.
 */
public class BatchFile {
  
  // Field length fixed by "pattern" attribute with value of type ".{0,N}"
  private static final String STRING_LENGTH_PATTERN = ".{0,";
  
  // Equivalent "codRegistro" values specified by "patterns" of type "0?NNNN" 
  private static final String EQUIVALENT_COD_REGISTRO_PATTERN = "0?";
  
  // XML name for "codRegistro" attribute (indicates type of record)
  private static final String COD_REGISTRO_XML_NAME = "codRegistro";

  /**
   * Write Castor object in file using fixed width records.
   * @param castorObject File content
   * @param file File for output
   * @throws Exception
   */
  public static void writeFile(Object castorObject, FileWriter file) throws Exception {
    
    // Castor object validity
    Method isValidMethod = castorObject.getClass().getMethod("isValid", new Class[]{});
    Boolean valid = (Boolean)isValidMethod.invoke(castorObject, new Object[]{});
    if (!valid) {
      Method marshal = castorObject.getClass().getMethod("marshal", new Class[]{Writer.class});
      marshal.invoke(castorObject, new Object[]{new StringWriter()});
    }
    
    // Write object into the file
    writeFileInternal(castorObject, file);
    file.endWrite();
    
  }
  
  /**
   * Write Castor object in file using fixed width records.
   * @param castorObject File content
   * @param file File for output
   * @throws Exception
   */
  @SuppressWarnings("unchecked")
  private static void writeFileInternal(Object castorObject, FileWriter file) throws Exception {
    
    LineWriter lineWriter = new LineWriter();

    // Castor object Descriptor
    String className = castorObject.getClass().getPackage().getName() + ".descriptors." + castorObject.getClass().getSimpleName() + "Descriptor";
    XMLClassDescriptorImpl descriptor = (XMLClassDescriptorImpl)Class.forName(className).newInstance();
    
    // Elements holded by this Castor object
    XMLFieldDescriptor[] fdList = (XMLFieldDescriptor[])descriptor.getFields();
    for (XMLFieldDescriptor fd : fdList) {
      
      // Complex element (simple and multiple elements)
      if (fd.getValidator().getTypeValidator() == null) {
        
        Method method = castorObject.getClass().getMethod("get" + fd.getXMLName(), new Class[]{});
        Object retrievedCastorObject = method.invoke(castorObject, new Object[]{});

        // Optional elements (minOccurs=0) should be not informed
        if (retrievedCastorObject != null) {
          if (retrievedCastorObject.getClass().isArray()) {
            Object[] array = (Object[]) retrievedCastorObject;
            for (Object element : array) {
              writeFileInternal(element, file);
            }
          } else {
            writeFileInternal(retrievedCastorObject, file);
          }
        }

      // Simple element
      } else {
        
        Method method = castorObject.getClass().getMethod("get" + fd.getXMLName().substring(0,1).toUpperCase() + fd.getXMLName().substring(1), new Class[]{});
        Object retrievedCastorObject = method.invoke(castorObject, new Object[]{});
        
        // Only numeric and string fields processed
        if (fd.getValidator().getTypeValidator() instanceof LongValidator) {
          
            // Length fixed by "totalDigits" attribute
            LongValidator lv = (LongValidator)fd.getValidator().getTypeValidator();
            lineWriter.addNumber(retrievedCastorObject, lv.getTotalDigits());
            
        } else {
          
          // Length fixed by "pattern" attribute of type ".{0,N}"
          StringValidator sv = (StringValidator)fd.getValidator().getTypeValidator();
          List patterns = sv.getPatterns();
          for(String pattern : patterns) {
            if (pattern.startsWith(STRING_LENGTH_PATTERN)) {
              String maxLength = pattern.substring(STRING_LENGTH_PATTERN.length(), pattern.length() - 1);
              lineWriter.addString(retrievedCastorObject, Integer.parseInt(maxLength));
            }
          }
          
        }
      }
    }

    if (lineWriter.getLine() != null && !"".equals(lineWriter.getLine())) {
      file.writeNext(lineWriter.getLine());
    }
    
  }
  
  /**
   * Returns Castor object filled with data from the file.
   * @param castorObject Initialized
   * @param file Reading file
   * @return Castor object containing file date
   * @throws Exception
   */
  public static Object readFile(Object castorObject, FileReader file) throws Exception {
    
    Object retCastorObject = readFileInternal(castorObject, file, null, null, null, false);
    file.endRead();

    // Castor object validity
    Method isValidMethod = retCastorObject.getClass().getMethod("isValid", new Class[]{});
    Boolean valid = (Boolean)isValidMethod.invoke(retCastorObject, new Object[]{});
    if (!valid) {
      Method marshal = retCastorObject.getClass().getMethod("marshal", new Class[]{Writer.class});
      marshal.invoke(retCastorObject, new Object[]{new StringWriter()});
    }
    return retCastorObject;
  }
  
  /**
   * Equivalent codes are specified in XSD by "patterns" of type "0?NNNN" 
   */
  private static Map<string , List> equivalentCodesMap = new HashMap<string , List>();
  private static Integer checkEquivalences(Integer codRegistro, Integer codRegistroPrevious) {
    for (Map.Entry<string , List> equivalentCodes : equivalentCodesMap.entrySet()) {
      if (equivalentCodes.getValue().contains(codRegistro) && equivalentCodes.getValue().contains(codRegistroPrevious)) {
        return codRegistro;
      }
    }
    return codRegistroPrevious;
  }
  
  private static List elementsInProcess = new ArrayList();
  
  /**
   * Returns Castor object filled with data from the file.
   * @param castorObject Object in use
   * @param file Reading file
   * @param lineReader Current line
   * @param codRegistroPrevious "codRegistro" previously read
   * @param parentCastorObject Parent object for object in use
   * @param multiple Element multiple ("maxOccurs" specified in XSD)
   * @return Castor object containing file date
   * @throws Exception
   */
  @SuppressWarnings("unchecked")
  private static Object readFileInternal(
      Object castorObject, 
      FileReader file, 
      LineReader lineReader,
      Integer codRegistroPrevious,
      Object parentCastorObject,
      boolean multiple) throws Exception {
    
    // Extract "codRegistro" from current line
    Integer codRegistro = null;
    if (lineReader == null || lineReader.isEnded()) {
      lineReader = new LineReader(file.readNext());
            codRegistro = lineReader.getInteger(5);
            lineReader.reset();
    } else if (lineReader.getCurrentPosition() == 0){
      codRegistro = lineReader.getInteger(5);
      lineReader.reset();
    }
    
    // Equivalent codes transformation
    codRegistroPrevious = checkEquivalences(codRegistro, codRegistroPrevious);
    
    // Elements in process (recurisivity control)
    if (elementsInProcess.contains(castorObject.getClass().getSimpleName())) {
      if (codRegistro == null && codRegistroPrevious == null) {
        codRegistroPrevious = -1;
      }
    }
    elementsInProcess.add(castorObject.getClass().getSimpleName());
    
    // Process elements from reading file
    if (codRegistroPrevious == null || (codRegistro != null && codRegistro.equals(codRegistroPrevious))) {
      
      String targetMultipleMethod = null;
      
      // Descriptor XML for current Castor object
      XMLClassDescriptorImpl descriptor = null;
      
      if (multiple && codRegistro != null && codRegistro.equals(codRegistroPrevious)) {
          String className = parentCastorObject.getClass().getPackage().getName() + ".descriptors." + parentCastorObject.getClass().getSimpleName() + "Descriptor";
          descriptor = (XMLClassDescriptorImpl)Class.forName(className).newInstance();
          targetMultipleMethod = "add" + castorObject.getClass().getSimpleName();
          castorObject = parentCastorObject;
      } else {
          String className = castorObject.getClass().getPackage().getName() + ".descriptors." + castorObject.getClass().getSimpleName() + "Descriptor";
          descriptor = (XMLClassDescriptorImpl)Class.forName(className).newInstance();
      }
      
      // Elements contained in Castor object
      XMLFieldDescriptor[] fdList = (XMLFieldDescriptor[])descriptor.getFields();
      for (XMLFieldDescriptor fd : fdList) {

        // Complex element (simple and multiple)
        if (fd.getValidator().getTypeValidator() == null) {
          
          Method[] methods = castorObject.getClass().getDeclaredMethods();
          
          // Search "add" or "set" method to create new instance 
          for (Method method : methods) {
            if (targetMultipleMethod == null || targetMultipleMethod.equals(method.getName())) {
              if (method.getName().equals("add" + fd.getXMLName())) {
                Class[] parameters = method.getParameterTypes();
                Object[] values = new Object[parameters.length];
                int i = 0;
                for(Class parameter : parameters) {
                  values[i] = parameter.newInstance();
                  i++;
                }
                method.invoke(castorObject, values);
                readFileInternal(values[0], file, lineReader, null, castorObject, true);
                break;
              } else if (method.getName().equals("set" + fd.getXMLName())) {
                Class[] parameters = method.getParameterTypes();
                Object[] values = new Object[parameters.length];
                int i = 0;
                for(Class parameter : parameters) {
                  values[i] = parameter.newInstance();
                  i++;
                }
                method.invoke(castorObject, values);
                readFileInternal(values[0], file, lineReader, null, castorObject, false);
                break;
              }
            }
          }
          
        // Simple element
        } else {
          
          // Detect equivalent codes
          if (fd.getXMLName().equals(COD_REGISTRO_XML_NAME) && equivalentCodesMap.get(castorObject.getClass().getSimpleName()) == null) {
            List equivalentCodes = new ArrayList();
              LongValidator lv = (LongValidator)fd.getValidator().getTypeValidator();
            List patterns = lv.getPatterns();
            for(String pattern : patterns) {
              if (pattern.startsWith(EQUIVALENT_COD_REGISTRO_PATTERN)) {
                equivalentCodes.add(Integer.parseInt(pattern.substring(2)));
              }
            }
            if (equivalentCodes.size() > 0) {
              equivalentCodesMap.put(castorObject.getClass().getSimpleName(), equivalentCodes);
            }
          }

          // Search "set" method to set the value
          Method targetMethod = null;
          Method[] methods = castorObject.getClass().getDeclaredMethods();
          for(Method method : methods) {
            if (method.getName().equals("set" + fd.getXMLName().substring(0,1).toUpperCase() + fd.getXMLName().substring(1))) {
              targetMethod = method;
              break;
            }
          }
          try {
            if (fd.getValidator().getTypeValidator() instanceof LongValidator) {
                LongValidator lv = (LongValidator)fd.getValidator().getTypeValidator();
                targetMethod.invoke(castorObject, new Object[]{lineReader.getLong(lv.getTotalDigits())});
            } else {
              StringValidator sv = (StringValidator)fd.getValidator().getTypeValidator();
              List patterns = sv.getPatterns();
              for(String pattern : patterns) {
                if (pattern.startsWith(STRING_LENGTH_PATTERN)) {
                  Integer maxLength = Integer.parseInt(pattern.substring(STRING_LENGTH_PATTERN.length(), pattern.length() - 1));
                  targetMethod.invoke(castorObject, new Object[]{lineReader.getString(maxLength)});
                }
              }
            }
          } catch (Exception e) {
              // Skip current element
              file.gotoPosition(file.getPreviousPosition());
              // Remove current Castor object
              if (multiple) {
                  String removeMethod = "remove" + castorObject.getClass().getSimpleName();
                  Method[] methodsParent = parentCastorObject.getClass().getDeclaredMethods();
                  for(Method method : methodsParent) {
                      if (method.getName().equals(removeMethod)) {
                          method.invoke(parentCastorObject, castorObject);
                      }
                  }
              } else {
                  String removeMethod = "set" + castorObject.getClass().getSimpleName();
                  Method[] methodsParent = parentCastorObject.getClass().getDeclaredMethods();
                  for(Method method : methodsParent) {
                      if (method.getName().equals(removeMethod)) {
                          method.invoke(parentCastorObject, new Object[] {null});
                      }
                   }
               }
            multiple = false;
            break;
          }
        }
      }
      
      // Search new instances
      if (multiple) {
        readFileInternal(castorObject, file, lineReader, codRegistro, parentCastorObject, true);
      }
      
    } else {
      // Skip element
      file.gotoPosition(file.getPreviousPosition());
    }
    
    // Clear empty objects
    int getMethodsCount = 0;
    int nullGetMethodsCount = 0;
    boolean countMethodIsZero = false;
    Method[] methods = castorObject.getClass().getDeclaredMethods();
    for (Method method : methods) {
      if (method.getName().startsWith("get") && method.getParameterTypes().length == 0) {
        getMethodsCount++;
        Object ret = method.invoke(castorObject);
        if (ret == null) {
          nullGetMethodsCount++;
        } else if (ret.getClass().isArray()) {
          Object[] o = (Object[]) ret;
          if (o.length == 0) {
            nullGetMethodsCount++;
          }
        } else if (ret instanceof Number) {
          Number n = (Number) ret;
          if (n.longValue() == 0) {
            nullGetMethodsCount++;
            countMethodIsZero = true;
          }
        }
      }
    }
    // Simple node with element list
    if (getMethodsCount == 2 && countMethodIsZero) {
      nullGetMethodsCount = getMethodsCount;
    }
    if (nullGetMethodsCount == getMethodsCount) {
      boolean removeMethodFound = false;
      String removeMethod = "remove" + castorObject.getClass().getSimpleName();
      Method[] methodsParent = parentCastorObject.getClass().getDeclaredMethods();
      for(Method method : methodsParent) {
        if (method.getName().equals(removeMethod)) {
          method.invoke(parentCastorObject, castorObject);
          removeMethodFound = true;
        }
      }
      if (!removeMethodFound) {
        removeMethod = "set" + castorObject.getClass().getSimpleName();
        methodsParent = parentCastorObject.getClass().getDeclaredMethods();
        for(Method method : methodsParent) {
          if (method.getName().equals(removeMethod)) {
            method.invoke(parentCastorObject, new Object[]{null});
          }
        }
      }
    }
    
    elementsInProcess.remove(castorObject.getClass().getSimpleName());
      
    return castorObject;
    
  }

}

5. Sample read and write code

public class TestCDWRecResFile {

  public static void main(String... args) throws Exception {
    writeCDWRecResFile();
    readCDWRecResFile();
  }

  private static Respuesta addRespuesta(String idSubTipoMensaje, String textoXML, int numLineasXML) {

    Respuesta respuesta = new Respuesta();

    CabeceraRespuesta cabeceraRespuesta = new CabeceraRespuesta();
    cabeceraRespuesta.setIdTipoMensaje("RES");
    cabeceraRespuesta.setIdSubTipoMensaje(idSubTipoMensaje);
    cabeceraRespuesta.setIdSolicitud("1");
    cabeceraRespuesta.setIdCartaCredito("1");
    cabeceraRespuesta.setIdCliente("1");
    cabeceraRespuesta.setFechaSolicitud("27032009");
    respuesta.setCabeceraRespuesta(cabeceraRespuesta);

    for(int i = 0; i < numLineasXML; i++) {
      DetalleRespuesta detalleRespuesta = new DetalleRespuesta();
      detalleRespuesta.setNumLinea((long)i + 1);
      detalleRespuesta.setTextoXML(textoXML);
      respuesta.addDetalleRespuesta(detalleRespuesta);
    }

    PieRespuesta pieRespuesta = new PieRespuesta();
    pieRespuesta.setNumLineas((long)(1 + respuesta.getDetalleRespuestaCount()  + 1));
    pieRespuesta.setNumLineasXML((long)respuesta.getDetalleRespuestaCount());
    respuesta.setPieRespuesta(pieRespuesta);

    return respuesta;

  }

  public static void writeCDWRecResFile() throws Exception {

    CDWRecResFile cdwRecResFile = new CDWRecResFile();

    CabeceraFichero cabeceraFichero = new CabeceraFichero();
    cabeceraFichero.setIdAplicacion("1");
    cabeceraFichero.setFechaProceso("27032009");
    cdwRecResFile.setCabeceraFichero(cabeceraFichero);

    cdwRecResFile.addRespuesta(addRespuesta("ACP", "Cosa1", 2));
    cdwRecResFile.addRespuesta(addRespuesta("REX", null, 0));

    PieFichero pieFichero = new PieFichero();
    pieFichero.setNumRespuestas((long)cdwRecResFile.getRespuestaCount());
    long numRegistros = 2l; // cabecera fichero + pie fichero
    Respuesta[] respuestas = cdwRecResFile.getRespuesta();
    for(Respuesta r : respuestas) {
      numRegistros = numRegistros + r.getPieRespuesta().getNumLineas();
    }
    pieFichero.setNumRegistros(numRegistros);
    cdwRecResFile.setPieFichero(pieFichero);

    FileWriter fw = new FileWriter("D:/apps/cdw/", "cdwrecres.txt");
    BatchFile.writeFile(cdwRecResFile, fw);

  }

  public static void readCDWRecResFile() throws Exception {

    FileReader file = new FileReader("D:/apps/cdw/", "cdwrecres.txt");

    CDWRecResFile cdwRecResFile = new CDWRecResFile();
    cdwRecResFile = (CDWRecResFile)BatchFile.readFile(cdwRecResFile, file);
    StringWriter sw = new StringWriter();
    cdwRecResFile.marshal(sw);
    System.out.println(sw.toString());

  }

As it can be seen, maintenance operations can be performed clearly on XSD file and the impact on Java code (sample on point 5) is reduced and can be controlled.

Some conventions have been asumed to develop this code:

  • codRegistro field indicates the type of record
  • Multiple elements can be repeated only if codRegistro field is the same for each line
  • Some Castor limitations provide a limited type XSD definition (p.e. the use of “pattern” restriction in String instead of  “length”)

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s