从 IBM Domino 检索大数据

Question

我正在尝试从 IBM Domino v9.0 收集帐户数据和组数据。

我为收集数据编写的代码使用了 lotus.domino.View API。该代码适用于小型数据集，我为 40 个用户 运行了该代码，结果很好。当我运行代码为 10k 用户 提取数据时，提取数据大约需要 3-4 小时，这是不可接受的。

您能否向我推荐一些其他 API，我们可以使用这些 API 以更快的速度检索数据？

我目前使用的代码如下：

private static void testUserDataRetrieval(String host, String userName, String password) throws Exception{
    Session s = NotesFactory.createSession(host, userName, password);
    Database db = s.getDatabase(s.getServerName(), "names");
    File outFile = new File("C:\\Users\\vishva\\Desktop\\dominoExtract.txt");
    FileWriter writer = new FileWriter(outFile);
    View view = db.getView("($Users)");
    ViewEntryCollection entryCollection = view.getAllEntries();
    writer.write("\n--------------------------------- Printing data for view:"+view.getName()+"("+view.getEntryCount()+")--------------------------------------------");
    Vector<Object> columnNames = view.getColumnNames();
    for(int i = 0; i< entryCollection.getCount(); i++){
        ViewEntry entry = entryCollection.getNextEntry();
        if(entry == null) continue;
        Vector<Object> colValues = entry.getColumnValues();
        for(int j = 0; j < columnNames.size(); j++)
            writer.write("\n"+columnNames.get(j)+":"+colValues.get(j));
        writer.write("\n*****************************************");
    }
    writer.flush();
    writer.close();
}

请告诉我应该使用哪些其他 API 来提高数据检索的速度？

Answer 1

感谢大家的意见。 为了检索用户，我稍后尝试了 LDAP API，它就像魅力一样工作。我能够在 23 秒 内检索到 10k 个帐户，这是可以接受的。我使用的代码如下

    private void extractDominoDataViaLdap(){
    String [] args = new String[]{"192.168.21.156","Administrator","password","","C:\\Users\\vishva\\Desktop"};
    String server = args[0];
    String userName = args[1];
    String password = args[2];
    String baseDN = args[3];
    String fileLocation = args[4];
    // Set up environment for creating initial context
    Hashtable<String, Object> env = new Hashtable<String, Object>(11);
    env.put(Context.INITIAL_CONTEXT_FACTORY,"com.sun.jndi.ldap.LdapCtxFactory");
    env.put(Context.PROVIDER_URL, "ldap://"+server+":389");

    long time = System.currentTimeMillis();
    // Authenticate as S. User and password "mysecret"
    env.put(Context.SECURITY_AUTHENTICATION, "simple");
    env.put(Context.SECURITY_PRINCIPAL, userName);
    env.put(Context.SECURITY_CREDENTIALS, password);
    FileWriter writer = null;
    BufferedWriter out = null;
    try {
        DirContext ctx = new InitialDirContext(env);
        //fetching user data
        // Create the search controls
        SearchControls searchCtls = new SearchControls();
        // Specify the attributes to return
        String returnedAtts[] = {"FullName","dn","displayname","givenName","sn","location","mail","mailfile","mailserver"};
        searchCtls.setReturningAttributes(returnedAtts);
        // Specify the search scope
        searchCtls.setSearchScope(SearchControls.SUBTREE_SCOPE);
        searchCtls.setCountLimit(0);
        // specify the LDAP search filter
        String searchFilter = "objectClass=inetOrgPerson";

        // Specify the Base for the search
        String searchBase = baseDN;

        // Search for objects using the filter
        NamingEnumeration<SearchResult> answer = ctx.search(searchBase,searchFilter, searchCtls);

        writer = new FileWriter(fileLocation+"\\users.csv");
        out = new BufferedWriter(writer);
        for(String attr : returnedAtts)
            out.write(attr+",");
        out.write("\n");
        int count = 0;
        // Loop through the search results
        while (answer.hasMoreElements()) {
            ++count;
            SearchResult sr = (SearchResult) answer.next();
            Attributes attrs = sr.getAttributes();
            StringBuilder sb = new StringBuilder();
            for(String attr : returnedAtts)
                sb.append("\""+((attrs.get(attr) != null)?attrs.get(attr).get():"")+"\"").append(",");
            out.write(sb.toString()+"\n");
            out.flush();
        }
        System.out.println("# of users returned: "+count);
        out.close();
        writer.close();
    }
}

Answer 2

默认情况下，您的 View 对象的 isAutoUpdate 属性设置为 true。这意味着它是synchronizing，其中包含后端发生的任何更改，而不仅仅是给您一次性快照。添加这个：

view.setAutoUpdate(false);

此外，您可能应该使用 ViewNavigator 类进行调查。请参阅 here 获取一篇文章，详细介绍自 Domino 8.5.2 起此类中的性能改进。它是由类的首席开发人员编写的。另请参阅blog post 了解很多有趣的细节。

Answer 3

首先：您在每次运行 for 循环时读取集合的计数。这使得它非常缓慢。 for 循环根本没有必要。第二：你从不回收这些物品。这在处理 domino 对象时是强制性的，否则您的代码会比您想象的更快地消耗内存：

private static void testUserDataRetrieval(String host, String userName, String password) throws Exception{
  Session s = NotesFactory.createSession(host, userName, password);
  Database db = s.getDatabase(s.getServerName(), "names");
  File outFile = new File("C:\\Users\\vishva\\Desktop\\dominoExtract.txt");
  FileWriter writer = new FileWriter(outFile);
  View view = db.getView("($Users)");
  ViewEntryCollection entryCollection = view.getAllEntries();
  writer.write("\n--------------------------------- Printing data for view:"+view.getName()+"("+view.getEntryCount()+")--------------------------------------------");
  Vector<Object> columnNames = view.getColumnNames();
  ViewEntry entry = entryCollection.getFirstEntry();
  ViewEntry entryNext;
  while(entry != null) {
    Vector<Object> colValues = entry.getColumnValues();
    for(int j = 0; j < columnNames.size(); j++)
        writer.write("\n"+columnNames.get(j)+":"+colValues.get(j));
    writer.write("\n*****************************************");
    entryNext = entryCollection.getNextEntry();
    entry.recycle();
    entry = entryNext ;
  }

writer.flush();
writer.close();

}