SecretsManagerClient looks up credentials using fs.readFile even if credentials are passed when initializing

[nwehner]

Describe the bug

Despite creating a new SecretsManagerClient with credentials included in the initialization options, the program uses fs.readFile to grab credentials off-disk. The SecretsManagerClient should be using the provided credentials and thus not use fs.readFile at all.

Your environment

aws/codebuild/standard:4.0 Linux image with npm@7.7.5 and node@12.18.0

SDK version number

@aws-sdk/client-s3@3.13.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Paste output of npx envinfo --browsers or node -v or react-native -v
v12.18.0

Steps to reproduce

If you create a new SecretsManagerClient like so
new SecretsManagerClient({ region: 'us-west-2', credentials: { accessKeyId: 'myAccessKey', secretAccessKey: 'mySecretKey' } })
and proceed to resolve the Promise, the slurpFile function from @aws-sdk/shared-ini-file-loader is used to grab credentials off the file system.

Observed behavior

While building a new Webpack project with SecretsManagerClient included therein, Webpack warns that there is "Unsafe builtin usage fs.readFile" pointing to the slurfFile function in @aws-sdk/shared-ini-file-loader/dist/es/index.js:87:17

Expected behavior

fs.readFile should not be invoked at all since credentials are passed to the client during initialization.

Additional context

This might possibly be related to #2027

[AllanZhengYP]

Hi @nwehner

fs.readFile exists in the SDK no matter whether you provide the credentials. Can you confirm it's acctually called in the credential resolution? because the there are a handful places where fs.readFile will be used. For example, SDK will try to load region from disk if not provided; SDK will also try loading the maxAttempt too if not provided:

aws-sdk-js-v3/clients/client-secrets-manager/runtimeConfig.ts

Line 31 in ca6f146

maxAttempts: loadNodeConfig(NODE_MAX_ATTEMPT_CONFIG_OPTIONS),

.

However, they will be read only once at construction.

[nwehner]

Good to know! Is there a list somewhere of all the properties that would need to be populated off-disk?

I made a small edit to the slurpFile function to reject the promise right away, rather than trying to readFile and resolving the promise with that data. When Gatsby's building everything with Webpack, I get an UnhandledPromiseRejectionWarning regarding the rejection I introduced. So, even with credentials and region populated in the SecretsManager options, the program is trying to get something off-disk.

[nwehner]

Did some more logging and I can confirm it's trying to grab my .aws/config file. That said, without the config file I can still retrieve all the secrets just fine, given that the SecretsManagerClient is initialized with region and credentials (accessKeyId & secretAccessKey).

[nwehner]

I tried initializing the SecretsManagerClient with every configuration option I could find, but the program is still trying to grab my .aws/config file...

{ apiVersion: '2017-10-17', region: 'us-west-2', credentials: { accessKeyId: getPublic(), secretAccessKey: getPrivate(), expiration: undefined, sessionToken: undefined }, maxAttempts: 10, tls: true, signer: undefined, signingEscapePath: undefined, systemClockOffset: undefined, signingRegion: undefined, retryStrategy: undefined, endpoint: undefined, customUserAgent: undefined, defaultUserAgentProvider: undefined, base64Decoder: undefined, base64Encoder: undefined, bodyLengthChecker: undefined, credentialDefaultProvider: undefined, streamCollector: undefined, disableHostPrefix: undefined, utf8Decoder: undefined, utf8Encoder: undefined, requestHandler: undefined, regionInfoProvider: undefined, serviceId: undefined, logger: undefined, urlParser: undefined, runtime: undefined, sha256: undefined }

[jamiehodge]

I believe we're seeing the same issue with the S3 client. Setting AWS_SDK_LOAD_CONFIG=0 has no effect.

[moltar]

This causes Lambda to fail on heavy load:

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at null.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

[TysonAndre-tmg]

Is the EMFILE still happening?

Starting in february 2022, after the previous comment, readFile's result was cached - it only performs the file read for a given file the first time that file path is requested (even if there are multiple concurrent requests for the same file). Before, it would call fs.readFile every time

https://github.com/aws/aws-sdk-js-v3/commits/main/packages/shared-ini-file-loader/src/slurpFile.ts

7b8954d#diff-322e53a7185a2b9b43ad5415ea1b6ffe0c5fd1f0858a58dae9d6b9b10d375a8d

https://github.com/aws/aws-sdk-js-v3/blob/main/packages/shared-ini-file-loader/src/getHomeDir.ts - it's still calling os.homedir() repeatedly, which is separate from the readFile bug. Is it safe to cache that (e.g. maybe conditionally on uid)?