SwiGLU Feed-Forward Layer

Problem

Implement the bias-free SwiGLU feed-forward sublayer: silu(x @ gate_weight) * (x @ up_weight), followed by multiplication by down_weight. Here silu(z) = z/(1+exp(−z)), and * means elementwise multiplication. Return the projected token rows; do not add a residual.

def swiglu(x, gate_weight, up_weight, down_weight):

Constraints

• x is T × D, gate_weight and up_weight D × H, down_weight H × O; dimensions are 1..12.
• All projected gate entries are in [-100,100].
• Inputs satisfy these constraints; invalid-input handling is not graded. Use only Python standard library and NumPy.

Examples

Arguments: [
  [
    [
      1
    ]
  ],
  [
    [
      0
    ]
  ],
  [
    [
      2
    ]
  ],
  [
    [
      3
    ]
  ]
]
Expected result: [
  [
    0
  ]
]

Arguments: [
  [
    [
      1
    ]
  ],
  [
    [
      1
    ]
  ],
  [
    [
      2
    ]
  ],
  [
    [
      3
    ]
  ]
]
Expected result: [
  [
    4.38635147178003
  ]
]