Apparently the exception doesn't occur if I use separate parent folder for
each thread, but the new documents SOMETIMES are created under incorrect
folder (created by other threads).
Attached is the source code but I cleaned up the code a little and took out
unrelated stuff of mine.
Please let me know the outcome ....
mike
On Fri, May 25, 2018 at 5:59 PM, Mignon, Laurent <laurent.mig...@acsone.eu>
wrote:
> Not sure if it's the root cause but new files are added at same time into
> the same parent folder. That means that the parent folder is updated at
> same time.
> Do you have the same problem if each thread create files in its own parent
> folder?
>
> lmignon
>
> On Fri, May 25, 2018 at 1:52 PM, Mike IT Expert <mikeitexp...@gmail.com>
> wrote:
>
> > You are right but my script does't update the same resource/document
> > simultaneously because each thread uploads new files under new unique
> names
> > on the server, in that each thread
> >
> > *1) creates its own instance of the repository*
> >
> > * # 2) upload files and update properties*
> > * # connect to the server *
> > * client = cmislib.CmisClient(SERVER_URL, USER, PASSWD)*
> >
> > * # get default repository*
> > * repo = client.defaultRepository*
> >
> > *2) retrieves the target folder's node id*
> >
> > * # get the target folder to upload new documents into*
> > * results = repo.query("select * from cmis:folder where cmis:name =
> '%s'"%(
> > TARGET_ALFRESCO_FOLDER, ))*
> > * targetFolder = results[0]*
> > * targetFolder = repo.getObject( targetFolder.id )*
> >
> > *3) for each file (eg. file_i ) in a loop python statement*
> >
> >
> >
> > * newDoc = targetFolder.createDocument( file_i['docName'], contentFile =
> > inFile ) *
> >
> > * props = {*
> > * u'cm:title': file_i[ HDR_TITLE ],*
> > * u'cm:description': file_i[ HDR_DESCR ],*
> > * }*
> >
> >
> > * newDoc.updateProperties(props)*
> >
> >
> > So what I am doing is that I access the same folder and upload new
> > documents using unique names. I am 100% sure the document names (ie.
> > * file_i['docName']*) are unique under the same folder.
> >
> > It just doesn't seem to me to be any concurrency problem with above code.
> >
> >
> > Many thanks anyways,
> > M
> >
> >
> > On Fri, May 25, 2018 at 3:57 PM, Mignon, Laurent <
> laurent.mig...@acsone.eu
> > >
> > wrote:
> >
> > > Hi Mike,
> > >
> > > I'm not aware of a design issue that could prevent to use cmislib
> python
> > > into multi threads.
> > > The problem you describe is a normal behaviour when two actors (human
> or
> > > prog) update the same resource at same time into alfresco.
> > >
> > > Regards,
> > >
> > > lmignon
> > >
> > > On Fri, May 25, 2018 at 12:19 PM, Mike IT Expert <
> mikeitexp...@gmail.com
> > >
> > > wrote:
> > >
> > >> Dear All,
> > >>
> > >> Apologies in advance for sending this message to dev mailing list (I
> > wish
> > >> there was userlist also I wasn't sure it is bug or not.)
> > >>
> > >> I have started working with cmislib since past week using Alfresco as
> > the
> > >> local server.
> > >>
> > >> My goal is to upload documents provided by json rcord list in a for
> > loop
> > >> as you see below *which is part of run-overiden method of
> > >> threading.Thread.run method for python 2.7*.
> > >>
> > >> The main code is attached just in case.
> > >>
> > >>
> > >>
> > >> The above code works find using ONE thread only. If I use more than
> one
> > >> thread I get the below exception for some records which is odd.
> > >>
> > >>
> > >> File "module_proj_2_parts_1_and_2.py", line 82, in run
> > >> newDoc = targetFolder.createDocument( docName, contentFile =
> inFile
> > )
> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.
> > >> 7/site-packages/cmislib/atompub/binding.py", line 2788, in
> > createDocument
> > >> contentEncoding)
> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.
> > >> 7/site-packages/cmislib/atompub/binding.py", line 1807, in
> > createDocument
> > >> ATOM_XML_ENTRY_TYPE)
> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.
> > >> 7/site-packages/cmislib/atompub/binding.py", line 188, in post
> > >> self._processCommonErrors(resp, url)
> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/python2.
> > >> 7/site-packages/cmislib/cmis_services.py", line 63, in
> > >> _processCommonErrors
> > >> raise UpdateConflictException(error[\'status\'],
> > >> url)\nUpdateConflictException: Error 409 at
> > >> http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/
> > >> versions/1.1/atom/children?id=4baddda1-87eb-45eb-8d0b-9958b8942c4f\n
> > >>
> > >>
> > >> I doubt if it is bug ... but if it is please let me know where I can
> > >> start to fix this.
> > >>
> > >> Also, my pip freeze package versions are as below:
> > >>
> > >> certifi==2018.4.16
> > >> chardet==3.0.4
> > >> cmislib==0.6.0
> > >> httplib2==0.11.3
> > >> idna==2.6
> > >> iso8601==0.1.12
> > >> requests==2.18.4
> > >> urllib3==1.22
> > >>
> > >> I appreciate any clue/comment.
> > >>
> > >> Best Regards,
> > >> Mike
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > > --
> > > *Laurent Mignon*
> > > Senior software engineer
> > >
> > > Tel: +352 20 21 10 20 32
> > > Fax: +352 20 21 10 21
> > > Email: laurent.mig...@acsone.eu
> > >
> > > Acsone SA, Succursale de Luxembourg
> > > 22, Zone Industrielle
> > >
> > > | L-8287 Kehlen, Luxembourg
> > > TVA LU24733605
> > > | RCS B160400
> > >
> > >
> > > Acsone sa/nv
> > > Waterloo Atrium, Drève Richelle 167
> > > <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167&entry=
> gmail&source=g
> > >
> > > | B-1410 Waterloo
> > > RPM Bruxelles 0835.207.216 RPR Brussel
> > >
> > > www.acsone.eu
> > >
> > >
> > >
> > >
> >
>
#! /usr/bin/env python
#
https://raw.githubusercontent.com/jpotts/alfresco-api-python-examples/master/cmis_create_document.py
import logging
from oauth2client.keyring_storage import Storage
from oauth2client.client import flow_from_clientsecrets
from oauth2client.tools import run
from httplib2 import Http
import json
from cmislib.model import CmisClient, CmisException
import os, pwd
NETWORK = 'alfresco.com'
SITE = 'alfresco-api-demo'
FOLDER_NAME = 'test folder'
FILE = '/home/mike/Desktop/hassan_proj2/git_sample1.py'
FILE_TYPE = 'application/txt'
uid = pwd.getpwuid(os.getuid())[0]
storage = Storage("Jeff's Sample Python App", uid)
http = Http(disable_ssl_certificate_validation=True) # Should not have to do
this!
flow = flow_from_clientsecrets('client_secrets.json',
scope='public_api',
redirect_uri='http://localhost:8080/')
credentials = storage.get()
if credentials == None:
credentials = run(flow, storage, http=http)
storage.put(credentials)
print "Your access_token is: %s" % credentials.access_token
http = credentials.authorize(http)
# Snag the user's home network
resp, content = http.request('https://api.alfresco.com')
networkList = json.loads(content)
homeNetwork = networkList['list']['entries'][0]['entry']['id']
print "Your home network appears to be: %s" % homeNetwork
headers = {'Authorization': 'Bearer ' + credentials.access_token}
client = CmisClient('https://api.alfresco.com/cmis/versions/1.0/atom',
'',
'',
headers=headers)
repo = client.defaultRepository
# Find the root folder of our target site
resp, content = http.request(
'https://api.alfresco.com/%s/public/alfresco/versions/1/sites/%s/containers' %
(NETWORK, SITE))
containerList = json.loads(content)
rootFolderId = containerList['list']['entries'][0]['entry']['id']
# Create a new folder in the root folder
rootFolder = repo.getObject(rootFolderId)
subFolder = None
try:
subFolder = rootFolder.createFolder(FOLDER_NAME)
print "Created folder: %s" % subFolder.id
except CmisException:
path = rootFolder.getPaths()[0]
subFolder = repo.getObjectByPath(path + '/' + FOLDER_NAME)
print "Folder already existed"
# Like the folder
body = '{"id": "likes", "myRating": true}'
resp, content = http.request(
'https://api.alfresco.com/%s/public/alfresco/versions/1/nodes/%s/ratings' %
(NETWORK, subFolder.id),
method='post',
body=body)
print "You liked: %s" % subFolder.id
# Create a test document in the subFolder
file = open(FILE, 'rb')
fileName = file.name.split('/')[-1]
doc = None
try:
doc = subFolder.createDocument(fileName, contentFile=file,
contentType=FILE_TYPE)
print "Created document: %s" % doc.id
except CmisException:
path = subFolder.getPaths()[0]
doc = repo.getObjectByPath(path + '/' + fileName)
print "Document already existed: %s" % fileName
file.close()
# Create a comment on the test document
# NOTE: When dealing with documents, the REST API wants a versionSeriesID!
body = '{"content": "Here is a comment!"}'
resp, content = http.request(
'https://api.alfresco.com/%s/public/alfresco/versions/1/nodes/%s/comments' %
(NETWORK, doc.properties['cmis:versionSeriesId']),
method='post',
body=body)
print "You commented on: %s" % doc.idimport os
import threading as trd
import json
import time
import random as rnd
import cmislib
import uuid
import traceback
FOLDER_PREFIX = 'EEE_'
USER = 'admin'
PASSWD = 'blabla123'
SERVER_NAME = '127.0.0.1'
SERVER_PORT = '8080'
LOG_FILE = 'log.txt'
SERVER_URL = 'http://%s:%s/alfresco/api/-default-/public/cmis/versions/1.1/atom'%(SERVER_NAME, SERVER_PORT)
class ToJsonConverterThread(trd.Thread):
'''
Takes a filepath and convert the records between start and
end positions into json record
'''
def __init__(self, threadId):
trd.Thread.__init__( self )
self.threadId = threadId
self.unprocessedRecords = []
def run(self):
'''
creates a new folder 'folder_by_thread_i' then
creates 5 documents under it
'''
# 2) upload files and update properties
# connect to the server
client = cmislib.CmisClient(SERVER_URL, USER, PASSWD)
# get default repository
repo = client.defaultRepository
# create a new folder for each thread
root = repo.rootFolder
targetFolder = root.createFolder( FOLDER_PREFIX + '__%s'%str(self.threadId) )
i = 0
for j in range(10):
# uploadFilename = IMAGE_INPUT_DIR + record['img']
uploadFilename = 'file1.txt'
with open(uploadFilename, 'rb') as inFile:
# upload file
try:
docName = "new_doc_by_thread_%s_no_%s.txt"%( str(self.threadId), str(j))
newDoc = targetFolder.createDocument( docName, contentFile = inFile )
print("File uploaded successfuly!")
# update properties
props = {
u'cm:title': 'title for doc %s by thread %s'%(docName, str(self.threadId)),
u'cm:description': 'description for doc %s by thread %s'%(docName, str(self.threadId)),
}
# newDoc.updateProperties(props)
except Exception as e:
record = {'threadId': self.threadId, }
record['stacktrace'] = traceback.format_exc()
self.unprocessedRecords.append(record)
print(traceback.format_exc())
finally:
pass
i += 1
# start the threads
nThread = 20
threadList = []
i = 0
while i < nThread:
# create a thread
thrd = ToJsonConverterThread( i )
thrd.start()
threadList.append( thrd )
i += 1
# wait for all threads to finish
allRecords = []
for thrd in threadList:
thrd.join()
# write the failed records to log file.
with open(LOG_FILE, 'w') as logFile:
for thrd in threadList:
for record in thrd.unprocessedRecords:
logFile.write( str( record ) + "\n\n")
