Recent comments posted to this site:
Another try - this time a slightly simpler setup using my version of the walkthrough commands
[[!format bash """
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive drop usbdrive walkthrough.sh ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr move walkthrough.sh (gpg) (checking flickr...) (to flickr...) /home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: /home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis whereis walkthrough.sh (1 copy) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) ok whereis walkthrough.sh~ (3 copies) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) 7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop) d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive) ok nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh get walkthrough.sh (from flickr...) (gpg) git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory) gpg: decrypt_message failed: eof
Unable to access these remotes: flickr
Try making some of these repositories available: 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log) Based on the location log, walkthrough.sh was expected to be present, but its content is missing.
No known copies exist of walkthrough.sh failed fsck walkthrough.sh~ (checking flickr...) (fixing location log) Based on the location log, walkthrough.sh~ ** was expected to be present, but its content is missing. failed (Recording state in git...) git-annex: fsck: 2 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$
""" ]]
Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex.
I copied one file to Flickr and dropped it elsewhere (--force). I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote.
Then I find I can't get the file back, even though there is a record of it from whereis.
Can you help enlighten me as to what am I missing? I assumed whereis would only report files that exist and can be copied back. If not my error, I can raise bug or search for logs. Thanks in advance for any help.
[[!format perl """
nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis whereis libpeerconnection.log (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis test.cgi (1 copy) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) ok whereis walkthrough.sh (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis walkthrough.sh~ (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi get test.cgi (from flickr...)
git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/tmp$
"""]]
The behavior you show with fsck --from is that the first time it's run against the damaged remote it notices the file is not present using the checkpresent hook. It then updates the location log. The subsequent times it's run, it sees that the location log says the file is not present in the remote. It verifies this is the case by calling the checkpresent hook. Since the two data sources agree, and numcopies is still satisfied, it prints "ok". There does not seem to be a bug here.
(return in Haskell does not do what you would expect to happen in a traditional imperative language. It does not alter control flow, and any function using return can be mechanically converted to one that does not use return.)
In the current HEAD the "checkpresent" method in Hook.hs is missing a "return" while other hooks have a return value eg. Directory.hs, Rsync.hs, ...
I noticed that if my checkpresent hook does not output anything (to be exact: I commented out all the lines in the block) it behaves strangely. I copied the test file to "copyrepo", removed it by hand (so git-annex does not know about the change) and executed a fsck.
~/annex5/test1/123 $ git annex fsck --from copyrepo --fast
fsck somefile.1 (checking copyrepo...) (fixing location log)
** Based on the location log, somefile.1
** was expected to be present, but its content is missing.
failed
running the check again
~/annex5/test1/123 $ ga fsck --from copyrepo --fast
fsck somefile.1 (checking copyrepo...) ok
and running the check again without --fast
~/annex5/test1/123 $ ga fsck --from copyrepo
fsck somefile.1 (checking copyrepo...) ok
It thinks, the file is in the repo but it is not.
If I were you I'd suppress that "File not found" error.
Hook special remotes can output messages to stderr, and it's also fine to output eg, progress bars to stdout when seding/receving files. But unnecessary cluttery output should be avoided.
Roger that.
If this is acceptable: terminal output screenshot, than I'm almost done and will publish soon. (Of course a REST API using client would much be better, but this is just the start.)
The checkpresent hook should always exit 0 unless there was an exceptional condition (eg, perhaps it cannot check if the file exists one way or the other). Like the documentation for it says, the important thing is what it outputs to stdout, which should contain the key name if it's present, and should not contain the key name if it's not present.
I hope you post to this website about your special remote when you get it fully working!
With debug turned on.
[[!format bash """
initremote flickr (encryption setup) (shared cipher) ok (Recording state in git...) describe flickr ok (Recording state in git...) /home/nrb/repos/annex/laptop-annex fsck walkthrough.sh (checksum...) ok fsck walkthrough.sh~ (checksum...) ok /home/nrb/repos/annex/laptop-annex copy walkthrough.sh (gpg) (checking flickr...) 16:18:52 [flickrannex-0.1.5] : 'START' 16:18:52 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:18:52 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:18:52 [flickrannex-0.1.5] readFile : 'Done' 16:18:52 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:18:54 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:18:54 [flickrannex-0.1.5] main : 'Trying page: 1' 16:18:55 [flickrannex-0.1.5] main : 'Error. found nothing:{'pages': '1', 'cancreate': '1', 'total': '0', 'page': '1', 'perpage': '0'}' 16:18:55 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 - u'gitannex' - '8086216@N08'' 16:18:55 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists' 16:18:55 [flickrannex-0.1.5] : 'STOP: 2s' (to flickr...) 16:18:55 [flickrannex-0.1.5] : 'START' 16:18:55 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P ANNEX_FILE=/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:18:55 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:18:55 [flickrannex-0.1.5] readFile : 'Done' 16:18:55 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:18:57 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:18:58 [flickrannex-0.1.5] main : 'Trying page: 1' 16:18:58 [flickrannex-0.1.5] main : 'Error. found nothing:{'pages': '1', 'cancreate': '1', 'total': '0', 'page': '1', 'perpage': '0'}' 16:18:58 [flickrannex-0.1.5] postFile : '/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 to u'gitannex' - GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0' 16:18:58 [flickrannex-0.1.5] postFile : 'pre /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 size: 1047 more than 40234050.' 16:18:58 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0' - 'rb'' 16:18:58 [flickrannex-0.1.5] readFile : 'Done' 16:18:58 [flickrannex-0.1.5] postFile : 'Uploading: /home/nrb/repos/gits/flickrannex/temp/encoded-GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0' /home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: /home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: 16:19:01 [flickrannex-0.1.5] postFile : 'Done: ' 16:19:01 [flickrannex-0.1.5] : 'STOP: 5s' ok copy walkthrough.sh~ (checking flickr...) 16:19:01 [flickrannex-0.1.5] : 'START' 16:19:01 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:19:01 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:19:01 [flickrannex-0.1.5] readFile : 'Done' 16:19:01 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:19:03 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:19:03 [flickrannex-0.1.5] main : 'Photoset gitannex found: ' 16:19:03 [flickrannex-0.1.5] main : 'Trying page: 1' 16:19:03 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 - 72157634897264995L - '8086216@N08'' 16:19:03 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists' 16:19:03 [flickrannex-0.1.5] : 'STOP: 1s' (to flickr...) 16:19:03 [flickrannex-0.1.5] : 'START' 16:19:03 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz ANNEX_FILE=/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:19:03 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:19:03 [flickrannex-0.1.5] readFile : 'Done' 16:19:03 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:19:05 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:19:05 [flickrannex-0.1.5] main : 'Photoset gitannex found: ' 16:19:05 [flickrannex-0.1.5] main : 'Trying page: 1' 16:19:05 [flickrannex-0.1.5] postFile : '/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 to 72157634897264995L - GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9' 16:19:05 [flickrannex-0.1.5] postFile : 'pre /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 size: 1044 more than 40234050.' 16:19:05 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9' - 'rb'' 16:19:05 [flickrannex-0.1.5] readFile : 'Done' 16:19:05 [flickrannex-0.1.5] postFile : 'Uploading: /home/nrb/repos/gits/flickrannex/temp/encoded-GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9' /home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: /home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: 16:19:08 [flickrannex-0.1.5] postFile : 'Done: ' 16:19:08 [flickrannex-0.1.5] : 'STOP: 4s' ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis whereis walkthrough.sh (3 copies) 86491ded-899c-425d-9470-bf446cb06db1 -- flickr (the flickr library) 8e766014-7154-4f4f-a04b-9d1b3d333db1 -- here (my laptop) eed7055b-743b-4ab6-a390-29cfd326005d -- usbdrive (portable USB drive) ok whereis walkthrough.sh~ (3 copies) 86491ded-899c-425d-9470-bf446cb06db1 -- flickr (the flickr library) 8e766014-7154-4f4f-a04b-9d1b3d333db1 -- here (my laptop) eed7055b-743b-4ab6-a390-29cfd326005d -- usbdrive (portable USB drive) ok nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr fsck walkthrough.sh (gpg) (checking flickr...) 16:22:57 [flickrannex-0.1.5] : 'START' 16:22:57 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:22:57 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:22:57 [flickrannex-0.1.5] readFile : 'Done' 16:22:57 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:22:58 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:22:59 [flickrannex-0.1.5] main : 'Photoset gitannex found: ' 16:22:59 [flickrannex-0.1.5] main : 'Trying page: 1' 16:22:59 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 - 72157634897264995L - '8086216@N08'' 16:22:59 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists' 16:22:59 [flickrannex-0.1.5] : 'STOP: 2s' (fixing location log) Based on the location log, walkthrough.sh was expected to be present, but its content is missing. failed fsck walkthrough.sh~ (checking flickr...) 16:22:59 [flickrannex-0.1.5] : 'START' 16:22:59 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr'' 16:22:59 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r'' 16:22:59 [flickrannex-0.1.5] readFile : 'Done' 16:22:59 [flickrannex-0.1.5] login : 'nrbray@yahoo.com' 16:23:01 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08'' 16:23:01 [flickrannex-0.1.5] main : 'Photoset gitannex found: ' 16:23:01 [flickrannex-0.1.5] main : 'Trying page: 1' 16:23:01 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 - 72157634897264995L - '8086216@N08'' 16:23:01 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists' 16:23:01 [flickrannex-0.1.5] : 'STOP: 1s' (fixing location log) Based on the location log, walkthrough.sh~ was expected to be present, but its content is missing. failed (Recording state in git...) git-annex: fsck: 2 failed
""" ]]