Use an embedding layer to access dictionary items based on the integer indices defined by the input tensor.
The following code shows how to access three dictionary items at indices `1`, `5`, and `2`. The
dictionary contains 7 items that are 5-element vectors, , therefore, the indices descriptor has the
shape (3), the dictionary descriptor has the shape (5, 7), and the output descriptor has the shape (5, 3).
let lookupIndices: [Int8] = [1, 5, 2]
let input = BNNSNDArrayDescriptor.allocate(initializingFrom: lookupIndices,
shape: .vector(lookupIndices.count))
// The dictionary contains 7 embeddings of 5-element vectors
let dictionaryData: [Float] = [ 01, 02, 03, 04, 05, // Dictionary item 0
11, 12, 13, 14, 15, // Dictionary item 1
21, 22, 23, 24, 25, // Dictionary item 2
31, 32, 33, 34, 35, // Dictionary item 3
41, 42, 43, 44, 45, // Dictionary item 4
51, 52, 53, 54, 55, // Dictionary item 5
61, 62, 63, 64, 65] // Dictionary item 6
let dictionaryItemSize = 5
let dictionaryItemCount = 7
let dictionary = BNNSNDArrayDescriptor.allocate(
initializingFrom: dictionaryData,
shape: .matrixLastMajor(dictionaryItemSize,
dictionaryItemCount))
let output = BNNSNDArrayDescriptor.allocateUninitialized(
scalarType: Float.self,
shape: .matrixLastMajor(dictionaryItemSize,
lookupIndices.count))
let layer = BNNS.EmbeddingLayer(input: input,
output: output,
dictionary: dictionary,
paddingIndex: 0,
maximumNorm: 0,
normType: .euclidean,
scalesGradientByFrequency: false)
try layer?.apply(batchSize: 1,
input: input,
output: output)
On return, the output tensor contains the folowing values:
[11.0, 12.0, 13.0, 14.0, 15.0,
51.0, 52.0, 53.0, 54.0, 55.0,
21.0, 22.0, 23.0, 24.0, 25.0]
The embedding layer supports clipping by maximum norm. The following code accesses the second and third items
from a dictionary that contains three 2 x 3 matrices. The code initializes the embedding layer with a maximum norm
that is the L2-norm of the first dictionary item.
let lookupIndices: [Int8] = [1, 2]
let input = BNNSNDArrayDescriptor.allocate(initializingFrom: lookupIndices,
shape: .vector(lookupIndices.count))
// The dictionary contains 3 embeddings on 2 x 3 matrices.
let dictionaryData: [Float] = [0.1, 0.2, 0.3,
0.4, 0.5, 0.6,
1, 3, 5,
2, 4, 6,
60, 50, 40,
30, 20, 10]
let dictionaryItemRowCount = 2
let dictionaryItemColumnCount = 3
let dictionaryItemCount = 3
let dictionary = BNNSNDArrayDescriptor.allocate(
initializingFrom: dictionaryData,
shape: .tensor3DLastMajor(dictionaryItemRowCount,
dictionaryItemColumnCount,
dictionaryItemCount))
let output = BNNSNDArrayDescriptor.allocateUninitialized(
scalarType: Float.self,
shape: .tensor3DLastMajor(dictionaryItemRowCount,
dictionaryItemColumnCount,
lookupIndices.count))
let maximumNorm: Float = 0.953939201416946 // L2-norm of first item in dictionary
let layer = BNNS.EmbeddingLayer(input: input,
output: output,
dictionary: dictionary,
paddingIndex: 0,
maximumNorm: maximumNorm,
normType: .euclidean,
scalesGradientByFrequency: false)
try layer?.apply(batchSize: 1,
input: input,
output: output)
On return, the output array descriptor contains the following values:
[0.1, 0.3, 0.5,
0.2, 0.4, 0.6,
0.6, 0.5, 0.4,
0.3, 0.2, 0.1]
Applies the layer backward to generate weights gradients.
- Parameter batchSize: The number of input-output pairs to process.
- Parameter input: The input descriptor.
- Parameter output: The output descriptor.
- Parameter outputGradient: The output gradient descriptor.
- Parameter inputGradient: The input gradient descriptor.
- Parameter weightsGradient: The weights gradient descriptor.
Applies the layer to several pairs of (input, output) data.
- Parameter batchSize: The number of input-output pairs to process.
- Parameter input: The input descriptor.
- Parameter output: The output descriptor.
Creates a new embedding layer.
- Parameter input: The input descriptor.
- Parameter output: The output descriptor.
- Parameter dictionary: The dictionary array descriptor.
- Parameter paddingIndex: The padding index.
- Parameter maximumNorm: If nonzero, the operation renormalizes any vector with a norm greater than `maximumNorm` during forward lookups.
- Parameter normType:The type of norm used by the layer if `maximumNorm` is nonzero.
- Parameter scalesGradientByFrequency: If true, gradients calculated during the backward pass are scaled based on the number of occurrence of the corresponding index in the input.
- Parameter filterParameters: Runtime filter parameters.
- Returns: A new embedding layer on success or `nil` if the parameters are invalid.
