Providing content with a pin request

[Gozala]

We were evaluating protocol/web3-dev-team#58 in the of protocol/web3-dev-team#62 and the subject of "where is pinning service going to get content from" came up. Assumption is that pinning service will fetch content from ipfs network raises some concerns:

1. What if node that just added content is behind the NAT that pinning service can't punch through ?
2. What if the service is used from the web browser, in which case it is highly unlikely pinning service will be able to deal it ?
3. If all you want is to add some content to IPFS and get it pinned, spinning up a full IPFS node and waiting until content is fetched from it is an overkill.

I remember @lidel was telling me about de facto hack of encoding content in an identity hashed CID, which might overcome some of the above listed concerns but raises whole new ones:

1. Given that it is not specified how reasonable it is to expect that it would be even supported by a pinning service ?
2. Does that supposed to work with multi-blocks scenarios ?

   I thought that was a case, but more I think about it less sense it makes to me.

3. Do we have a CID size limit / request payload size limit to consider ?

Either way, uploading content as identity hashed CID encoded in base64 string in JSON feels like a very impractical solution to meet specific requirements. It seems like we need to consider extending this specification to support this use case or it will not be practical for cases where just putting content on IPFS is desired.

It is also worth pointing out here that e.g. pinata has own API for such a use case https://pinata.cloud/documentation#PinFileToIPFS

/cc @alanshaw @mikeal @jnthnvctr

[mikeal]

We should just PUT a CAR file with a single root (standard Filecoin CAR file). It’s pretty clean and we already have code to pin a CAR file in Go and the JS client libraries were just updated to be smaller and faster. We have nice library infrastructure to leverage here and we end up with some really thin client libraries you can build without IPFS in the client at all.

[aschmahmann]

This is a duplicate of ipfs/in-web-browsers#18 and ipfs/in-web-browsers#22 which are already closed and resolved.

Problems 1 and 2 are solved if you assume the remote pinning service is dialable which IMO is both fair and is a prerequisite for this solution to work.

Problem 3 I think is a non-issue. Given that you have a blockstore (FlatFS, badger, S3, a CAR file, etc.) spinning up a libp2p node that supports Bitswap and only makes a single connection to the pinning service node is not a huge ask and helps us plan for the future.

[mikeal]

spinning up a libp2p node that supports Bitswap and only makes a single connection to the pinning service node is not a huge ask and helps us plan for the future.

To a Web developer working in the browser, this is a huge ask.

[aschmahmann]

To a Web developer working in the browser, this is a huge ask.

I mean we could just provide a library that does this. If asking developers to include a library is a huge ask then I don't see how they get any real benefit out of IPFS, since they can't get the IPLD data/CAR file to send to the pinning service without using some library for working with IPLD data.

[mikeal]

Applications are a little more complicated than that. The client/user that needs to upload the content has different needs than the consumers of that data who need it available in a decentralized network.

In some NFT use cases an artist needs to put their content into a website and then never touch the site again. There are other users who might bid, buy, trade and do other authentication against the NFT data.

IPFS is still very much a critical part of this application, but the thing standing in the way of getting more content into IPFS is that loading content you’re just trying to hand to a remote provider to run IPFS for you requires such a substantial client.

[aschmahmann]

the thing standing in the way of getting more content into IPFS is that loading content you’re just trying to hand to a remote provider to run IPFS for you requires such a substantial client.

To be convinced of any value in this argument I'd need to see some evidence that the "substantial client" of a minimal libp2p client is much more burdensome than the IPLD (and likely UnixFS) libraries needed to process the data to the point that it would have a noticeable impact on a user's choices.

Additionally, while there are some nice aspects in terms of ease of implementation using HTTP to send the data to be pinned we have issues like ipfs/in-web-browsers#9 and ipfs/in-web-browsers#7 which become impossible when you start adding in HTTP specific features (e.g. PUT for a CAR file) into this spec. You also lose out on any deduplication benefits of a protocol like Bitswap or GraphSync.

Overall, IMO this proposal is pushing us in the wrong direction. If there is some major hurdle (e.g. with using js-libp2p in the browser) with the existing API's support for this use case, then I could potentially get behind this but otherwise I don't think it's worthwhile.

Curious if @lidel has a different perspective here.

[lidel]

Bit late to this party, but here are my condensed thoughts:

• "hack of encoding content in an identity hashed CID" is not feasible in real life. It works only for "helloworld"-like strings used in CI tests and was just a way to make mock-ipfs-pinning-service deterministic and run faster on CI (removing the data transfer step).
• Like @aschmahmann noted, data transfer from client behind NAT and other limiting network topologies is solved by provider hints in origins and delegates (both sides attempting to connect to each other using those addrs in best-effort fashion).
  • This is a simple but really powerful primitive: removes the need for any DHT lookup from the picture, and works out of the box when pin remote commands are used in IPFS node because it fills origins and pre-connects to delegates out-of-the-box.
  • Moreover, if pinning service returns /wss/ multiaddr in delegates then js-ipfs running on a web page will be able to connect to the service and start bitswap immediately. This means content routing is no longer an issue when pinning happens in the browser (assuming Remote pinning service implementation in JS-IPFS protocol/web3-dev-team#58 happens).
• Getting data into IPFS is out of scope of this API, so I am closing this issue.
  • This API operates on CIDs + optional provider hints and is not concerned by HOW data is imported to IPFS.
    • I believe that "import API" should integral part of HTTP Gateway.
      If you feel strongly about this, please propose specific mechanics in Proposal: IPFS Gateway HTTP API protocol/web3-dev-team#1
  • In practice, I see two solutions for importing data to IPFS:
    • Web app running full js-ipfs node and import happening via ipfs.add + pin.remote.add for true persistence and content routing (needs Remote pinning service implementation in JS-IPFS protocol/web3-dev-team#58)
    • Web app delegating import to some backend service running go-ipfs, and getting CID in return OR running js-ipfs-http-client in web app itself and doing pin.add with go-ipfs that way
  • "PUT a CAR file" or "PUT file to HTTP Gateway" would be the implementation detail that falls under one of above.
    • IMO only "HTTP PUT file to HTTP Gateway and get a CID" improves anything, because it does not require ANY library.
    • "Put a CAR" means I still need to run IPFS-aware chunker and CAR generator, which most likely won't be smaller than running full js-ipfs on a page.

[Gozala]

I have spend little more time thinking about this and I came to following conclusions that I would like get feedback on:

There are two very different user groups that are potential pinning service API users:

1. Users who run IPFS nodes and want things to be pinned remotely.
2. Users who just want to put content on IPFS.

I think current API does poor job at meeting either of those user needs. For group operating IPFS node libp2p based API would be a more effective and efficient avoiding lot of HTTP roundtrips etc...

For a group that just wants to add file to IPFS network having to run IPFS node just to add a file is a huge burden and sometimes constraints of the runtime also get in the way (e.g. serverless is a good example).

I think there is an opportunity to enable that second group and significantly reduce upfront costs (education, etc...) to get them into IPFS. I think doing it as part of pinning services is good idea because:

1. It makes services swapable, removing platform lock-in.
2. It make IPFS dead-simple choice for storing data.

[mikeal]

I’d like to pause this discussion for now.

It would seem that the entire purpose of a remote pinning API would be to hire a third party to run IPFS for you, and that effectively requiring a client to run IPFS in order to get content into that remote would be a barrier to maximizing the usefulness of the remote pinning service.

That said, you’re right to question how useful this would be and compare it to other work streams. I don’t think it would be a productive use of your time to send you all the NFT user research we’ve done and invite you to several more meetings where we’ve been discussing such things.

If we’re confident enough of the user need we can just build something to satisfy it ourselves. Once it’s deployed and used by these target users we can iterate on it and take what we’ve learned back to you and recommend changes to the pinning API with much more confidence in their usefulness. Then we can update anything we have already deployed to match what ends up being formally specified.

[lidel]

Revisiting: providing DAG archive with a pin request

We marinated a bit for a few weeks, and it seems that adding the ability to upload precomputed DAG archive to a pinning service is something that, if implemented in a thoughtful way, does not go over raison d'être of this spec. @rvagg makes a good point that as long we talk about DAG archive, there is no change in existing separation of concerns:

In the Pinning API we have the notion of "provider hints" and I'm wondering, [..] what the distinction might be between "go and find it over there and pin it" vs "here it is, just pin this" (i.e. the provider hint is essentially just "it's right here!").

"Uploading" DAG archive to pinning service is the ultimate "provider hint". It operates at the same abstraction level as bitswap session, does not introduce any complexity related to vs directory handling, chunking, hashes, which greatly simplifies things on the service end.

---

Prior art: DAG import APIs

Right now, we have two preexisting "DAG import" endpoints:

• go-ipfs: /api/v0/dag/import is provided on API port of every IPFS node
• ipfs-cluster: /add is /api/v0/add augmented with support for &format=car parameter

We want to make self-hosting of pinning services easier by adding support for this spec to ipfs-cluster (ipfs-cluster/ipfs-cluster#1213), but even when that happens, we will most likely need a separate namespace/port for "service" endpoints guarded by bearer access-token.

Requirements and Constraints

Main ones:

• Avoid magic: one should be able to use pinning service with generic HTTP clients like curl: do ipfs dag export, rsync archive to different machine, then pin remotely via curl
  • This is solved in prior art by sending DAG archive produced by ipfs dag export as multipart/form-data
• /pins supports pinning of a single CID at a time
  • Most likely means DAG import will be also limited to a single file upload
• Data from imported DAG should not get garbage collected before user pins it.
  At the same time, service should not pay the cost of keeping data that is not pinned by anyone.
  • We could require pinning service to protect manually imported data from GC for some time, but this shifts complexity to implementer (suddenly, one needs to track import time and expiration)
  • Better approach may be "import" operation that is atomic "import+pin"

Initial API idea (looking for feedback 👀 )

• To keep things easy for everyone (users, services) import+pin operation should be atomic.
  • Request is multipart/form-data (follow behavior of existing import endpoints)
  • Response returned after DAG import should be JSON with PinStatus object, same behavior as in existing "Add pin object" operation. PinStatus.status maps nicely:
    • If import was successful, the response should have status set to pinned immediately
    • If some blocks were missing from archive, it should be queued, and the service should try collecting them via DHT
    • If archive is invalid or pinning is not possible due to regular reasons, failed
  • Created pin is nameless, user can customize name/metadata via inexpensive "Replace pin object" operation, if needed (no need to complicate import operation).

Obligatory 🏚️ 🚲 question: how extensible we want this to be?

• (A) introduce /imports[/dag] endpoint that accepts multipart/form-data POST with a single DAG archive
  • This makes sense if we want to keep the door open for extending import capabilities in near future (multiple archives, maybe even directories and files like in /api/v0/add etc)
• (B) make /pins accept POST multipart/form-data
  • Pretty clean solution, works well with existing scope, but is not extensible, and based on my OpenAPI experience having more than one accepted content type may cause issues with docs and/or code generation.

(A) sounds like more pragmatic approach, YAML will produce good docs, we could extend it if we wanted, but lmk if I missed something important at any stage of this exploration.

cc @mikeal @olizilla @obo20 @hsanjuan @rvagg @ipfs/wg-pinning-services