Skip to content

Regularization implementation #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Regularization implementation #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions src/main/java/com/dj/core/model/graph/ConnectedNeuron.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,24 @@ public void backwardSignalReceived(final Double error) {

final var dzLearningRate = dz * context.getLearningRate();
backwardConnections = backwardConnections.add(inputSignals.scalarMultiply(dzLearningRate));
if (context.getRegularizationRate() != 0.) {
backwardConnections.walkInColumnOrder(new RealMatrixChangingVisitor() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@b0noI
backwardConnections is a 1:N vector.
So, in my view, it makes sense to use walkInRowOrder instead of walkInColumnOrder

@Override
public void start(final int i, final int i1, final int i2, final int i3, final int i4, final int i5) {

}

@Override
public double visit(final int i, final int i1, final double v) {
return v - Math.pow(v, context.getRegularizationLevel()) * context.getRegularizationRate();
}

@Override
public double end() {
return 0;
}
});
}

bias.addAndGet(inputSignalsAverage * dz * context.getLearningRate());
neuronIndexes
Expand Down
20 changes: 20 additions & 0 deletions src/main/java/com/dj/core/model/graph/Context.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ public class Context implements Serializable {

private boolean debugMode;

private int regularizationLevel = 2;

private double regularizationRate = 0.;

public Context(final double learningRate, final boolean debugMode) {
this.learningRate = learningRate;
this.debugMode = debugMode;
Expand All @@ -33,4 +37,20 @@ public boolean isDebugMode() {
public void setDebugMode(final boolean debugMode) {
this.debugMode = debugMode;
}

public int getRegularizationLevel() {
return regularizationLevel;
}

public void setRegularizationLevel(final int regularizationLevel) {
this.regularizationLevel = regularizationLevel;
}

public double getRegularizationRate() {
return regularizationRate;
}

public void setRegularizationRate(final double regularizationRate) {
this.regularizationRate = regularizationRate;
}
}