• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java AbstractStateBackend类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.flink.runtime.state.AbstractStateBackend的典型用法代码示例。如果您正苦于以下问题:Java AbstractStateBackend类的具体用法?Java AbstractStateBackend怎么用?Java AbstractStateBackend使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



AbstractStateBackend类属于org.apache.flink.runtime.state包,在下文中一共展示了AbstractStateBackend类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: parameters

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Parameterized.Parameters
public static Collection<AbstractStateBackend> parameters() throws IOException {
	TemporaryFolder tempFolder = new TemporaryFolder();
	tempFolder.create();

	MemoryStateBackend syncMemBackend = new MemoryStateBackend(MAX_MEM_STATE_SIZE, false);
	MemoryStateBackend asyncMemBackend = new MemoryStateBackend(MAX_MEM_STATE_SIZE, true);

	FsStateBackend syncFsBackend = new FsStateBackend("file://" + tempFolder.newFolder().getAbsolutePath(), false);
	FsStateBackend asyncFsBackend = new FsStateBackend("file://" + tempFolder.newFolder().getAbsolutePath(), true);

	RocksDBStateBackend fullRocksDbBackend = new RocksDBStateBackend(new MemoryStateBackend(MAX_MEM_STATE_SIZE), false);
	fullRocksDbBackend.setDbStoragePath(tempFolder.newFolder().getAbsolutePath());

	RocksDBStateBackend incRocksDbBackend = new RocksDBStateBackend(new MemoryStateBackend(MAX_MEM_STATE_SIZE), true);
	incRocksDbBackend.setDbStoragePath(tempFolder.newFolder().getAbsolutePath());

	return Arrays.asList(
		syncMemBackend,
		asyncMemBackend,
		syncFsBackend,
		asyncFsBackend,
		fullRocksDbBackend,
		incRocksDbBackend);
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:26,代码来源:PartitionedStateCheckpointingITCase.java


示例2: materialize

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Override
public StateHandle<DataInputView> materialize() throws Exception {
	AbstractStateBackend.CheckpointStateOutputView out = backend.createCheckpointStateOutputView(
			checkpointId,
			timestamp);

	int numWindows = stateSnapshot.size();
	out.writeInt(numWindows);

	for (Map.Entry<Long, Map<Long, Long>> window: stateSnapshot.entrySet()) {
		out.writeLong(window.getKey());
		int numKeys = window.getValue().size();
		out.writeInt(numKeys);

		for (Map.Entry<Long, Long> value : window.getValue().entrySet()) {
			out.writeLong(value.getKey());
			out.writeLong(value.getValue());
		}
	}

	return out.closeAndGetHandle();
}
 
开发者ID:dataArtisans,项目名称:query-window-example,代码行数:23,代码来源:QueryableWindowOperator.java


示例3: testCallsForwardedToNonPartitionedBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Test
public void testCallsForwardedToNonPartitionedBackend() throws Exception {
	AbstractStateBackend nonPartBackend = mock(AbstractStateBackend.class);
	RocksDBStateBackend rocksDbBackend = new RocksDBStateBackend(nonPartBackend);

	Environment env = getMockEnvironment();
	rocksDbBackend.createStreamFactory(env.getJobID(), "foobar");

	verify(nonPartBackend, times(1)).createStreamFactory(any(JobID.class), anyString());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:11,代码来源:RocksDBStateBackendConfigTest.java


示例4: setStateBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * @deprecated Use {@link #setStateBackend(StateBackend)} instead.
 */
@Deprecated
@PublicEvolving
public StreamExecutionEnvironment setStateBackend(AbstractStateBackend backend) {
	this.defaultStateBackend = Preconditions.checkNotNull(backend);
	return this;
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:10,代码来源:StreamExecutionEnvironment.java


示例5: DataInputViewAsynchronousStateHandle

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
public DataInputViewAsynchronousStateHandle(long checkpointId,
		long timestamp,
		Map<Long, Map<Long, Long>> stateSnapshot,
		AbstractStateBackend backend) {
	this.checkpointId = checkpointId;
	this.timestamp = timestamp;
	this.stateSnapshot = stateSnapshot;
	this.backend = backend;
}
 
开发者ID:dataArtisans,项目名称:query-window-example,代码行数:10,代码来源:QueryableWindowOperator.java


示例6: getStateBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * State backend to store Beam's state during computation.
 * Note: Only applicable when executing in streaming mode.
 */
@Description("Sets the state backend to use in streaming mode. "
    + "Otherwise the default is read from the Flink config.")
@JsonIgnore
AbstractStateBackend getStateBackend();
 
开发者ID:apache,项目名称:beam,代码行数:9,代码来源:FlinkPipelineOptions.java


示例7: createStateBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Override
protected AbstractStateBackend createStateBackend() throws Exception {
	return new FsStateBackend(temporaryFolder.newFolder().toURI().toString());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:5,代码来源:NonHAQueryableStateITCaseFsBackend.java


示例8: createStateBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Override
protected AbstractStateBackend createStateBackend() throws Exception {
	return new MemoryStateBackend();
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:5,代码来源:QueryableStateITCaseMemoryBackend.java


示例9: createStateBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
@Override
protected AbstractStateBackend createStateBackend() throws Exception {
	return new RocksDBStateBackend(temporaryFolder.newFolder().toURI().toString());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:5,代码来源:QueryableStateITCaseRocksDBBackend.java


示例10: testProgramWithBackend

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
protected void testProgramWithBackend(AbstractStateBackend stateBackend) throws Exception {
	assertEquals("Broken test setup", 0, (NUM_STRINGS / 2) % NUM_KEYS);

	final StreamExecutionEnvironment env = new TestStreamEnvironment(cluster, PARALLELISM);
	env.setParallelism(PARALLELISM);
	env.enableCheckpointing(500);
	env.getConfig().disableSysoutLogging();
	env.setRestartStrategy(RestartStrategies.fixedDelayRestart(Integer.MAX_VALUE, 0L));

	env.setStateBackend(stateBackend);

	// compute when (randomly) the failure should happen
	final int failurePosMin = (int) (0.6 * NUM_STRINGS / PARALLELISM);
	final int failurePosMax = (int) (0.8 * NUM_STRINGS / PARALLELISM);
	final int failurePos = (new Random().nextInt(failurePosMax - failurePosMin) + failurePosMin);

	final DataStream<Integer> stream1 = env.addSource(
			new IntGeneratingSourceFunction(NUM_STRINGS / 2, NUM_STRINGS / 4));

	final DataStream<Integer> stream2 = env.addSource(
			new IntGeneratingSourceFunction(NUM_STRINGS / 2, NUM_STRINGS / 4));

	stream1.union(stream2)
			.keyBy(new IdentityKeySelector<Integer>())
			.map(new OnceFailingPartitionedSum(failurePos))
			.keyBy(0)
			.addSink(new CounterSink());

	env.execute();

	// verify that we counted exactly right
	assertEquals(NUM_KEYS, CounterSink.ALL_COUNTS.size());
	assertEquals(NUM_KEYS, OnceFailingPartitionedSum.ALL_SUMS.size());

	for (Entry<Integer, Long> sum : OnceFailingPartitionedSum.ALL_SUMS.entrySet()) {
		assertEquals((long) sum.getKey() * NUM_STRINGS / NUM_KEYS, sum.getValue().longValue());
	}
	for (long count : CounterSink.ALL_COUNTS.values()) {
		assertEquals(NUM_STRINGS / NUM_KEYS, count);
	}
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:42,代码来源:KeyedStateCheckpointingITCase.java


示例11: testCleanupOfSnapshotsInFailureCase

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Test that the snapshot files are cleaned up in case of a failure during the snapshot
 * procedure.
 */
@Test
public void testCleanupOfSnapshotsInFailureCase() throws Exception {
	long checkpointId = 1L;
	long timestamp = 42L;

	Environment env = new DummyEnvironment("test task", 1, 0);

	CheckpointStreamFactory.CheckpointStateOutputStream outputStream = mock(CheckpointStreamFactory.CheckpointStateOutputStream.class);
	CheckpointStreamFactory checkpointStreamFactory = mock(CheckpointStreamFactory.class);
	AbstractStateBackend stateBackend = mock(AbstractStateBackend.class);

	final IOException testException = new IOException("Test exception");

	doReturn(checkpointStreamFactory).when(stateBackend).createStreamFactory(any(JobID.class), anyString());
	doThrow(testException).when(outputStream).write(anyInt());
	doReturn(outputStream).when(checkpointStreamFactory).createCheckpointStateOutputStream(eq(checkpointId), eq(timestamp));

	RocksDBStateBackend backend = new RocksDBStateBackend(stateBackend);

	backend.setDbStoragePath(temporaryFolder.newFolder().toURI().toString());

	AbstractKeyedStateBackend<Void> keyedStateBackend = backend.createKeyedStateBackend(
		env,
		new JobID(),
		"test operator",
		VoidSerializer.INSTANCE,
		1,
		new KeyGroupRange(0, 0),
		null);

	try {

		keyedStateBackend.restore(null);

		// register a state so that the state backend has to checkpoint something
		keyedStateBackend.getPartitionedState(
			"namespace",
			StringSerializer.INSTANCE,
			new ValueStateDescriptor<>("foobar", String.class));

		RunnableFuture<KeyedStateHandle> snapshotFuture = keyedStateBackend.snapshot(
			checkpointId, timestamp, checkpointStreamFactory, CheckpointOptions.forCheckpoint());

		try {
			FutureUtil.runIfNotDoneAndGet(snapshotFuture);
			fail("Expected an exception to be thrown here.");
		} catch (ExecutionException e) {
			Assert.assertEquals(testException, e.getCause());
		}

		verify(outputStream).close();
	} finally {
		IOUtils.closeQuietly(keyedStateBackend);
		keyedStateBackend.dispose();
	}
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:61,代码来源:RocksDBAsyncSnapshotTest.java


示例12: testQueryUnknownKey

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests the failure response with {@link UnknownKeyOrNamespace} as cause
 * on queries for non-existing keys.
 */
@Test
public void testQueryUnknownKey() throws Exception {
	KvStateRegistry registry = new KvStateRegistry();
	AtomicKvStateRequestStats stats = new AtomicKvStateRequestStats();

	KvStateServerHandler handler = new KvStateServerHandler(registry, TEST_THREAD_POOL, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	KeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<Integer> desc = new ValueStateDescriptor<>("any", IntSerializer.INSTANCE);
	desc.setQueryable("vanilla");

	backend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, desc);

	byte[] serializedKeyAndNamespace = KvStateRequestSerializer.serializeKeyAndNamespace(
			1238283,
			IntSerializer.INSTANCE,
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE);

	long requestId = Integer.MAX_VALUE + 22982L;

	assertTrue(registryListener.registrationName.equals("vanilla"));

	ByteBuf request = KvStateRequestSerializer.serializeKvStateRequest(
			channel.alloc(),
			requestId,
			registryListener.kvStateId,
			serializedKeyAndNamespace);

	// Write the request and wait for the response
	channel.writeInbound(request);

	ByteBuf buf = (ByteBuf) readInboundBlocking(channel);
	buf.skipBytes(4); // skip frame length

	// Verify the response
	assertEquals(KvStateRequestType.REQUEST_FAILURE, KvStateRequestSerializer.deserializeHeader(buf));
	KvStateRequestFailure response = KvStateRequestSerializer.deserializeKvStateRequestFailure(buf);

	assertEquals(requestId, response.getRequestId());

	assertTrue("Did not respond with expected failure cause", response.getCause() instanceof UnknownKeyOrNamespace);

	assertEquals(1, stats.getNumRequests());
	assertEquals(1, stats.getNumFailed());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:68,代码来源:KvStateServerHandlerTest.java


示例13: testQueryExecutorShutDown

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests the failure response on a rejected execution, because the query
 * executor has been closed.
 */
@Test
public void testQueryExecutorShutDown() throws Exception {
	KvStateRegistry registry = new KvStateRegistry();
	AtomicKvStateRequestStats stats = new AtomicKvStateRequestStats();

	ExecutorService closedExecutor = Executors.newSingleThreadExecutor();
	closedExecutor.shutdown();
	assertTrue(closedExecutor.isShutdown());

	KvStateServerHandler handler = new KvStateServerHandler(registry, closedExecutor, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	KeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<Integer> desc = new ValueStateDescriptor<>("any", IntSerializer.INSTANCE);
	desc.setQueryable("vanilla");

	backend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, desc);

	assertTrue(registryListener.registrationName.equals("vanilla"));

	ByteBuf request = KvStateRequestSerializer.serializeKvStateRequest(
			channel.alloc(),
			282872,
			registryListener.kvStateId,
			new byte[0]);

	// Write the request and wait for the response
	channel.writeInbound(request);

	ByteBuf buf = (ByteBuf) readInboundBlocking(channel);
	buf.skipBytes(4); // skip frame length

	// Verify the response
	assertEquals(KvStateRequestType.REQUEST_FAILURE, KvStateRequestSerializer.deserializeHeader(buf));
	KvStateRequestFailure response = KvStateRequestSerializer.deserializeKvStateRequestFailure(buf);

	assertTrue(response.getCause().getMessage().contains("RejectedExecutionException"));

	assertEquals(1, stats.getNumRequests());
	assertEquals(1, stats.getNumFailed());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:62,代码来源:KvStateServerHandlerTest.java


示例14: testChunkedResponse

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests that large responses are chunked.
 */
@Test
public void testChunkedResponse() throws Exception {
	KvStateRegistry registry = new KvStateRegistry();
	KvStateRequestStats stats = new AtomicKvStateRequestStats();

	KvStateServerHandler handler = new KvStateServerHandler(registry, TEST_THREAD_POOL, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	AbstractKeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<byte[]> desc = new ValueStateDescriptor<>("any", BytePrimitiveArraySerializer.INSTANCE);
	desc.setQueryable("vanilla");

	ValueState<byte[]> state = backend.getPartitionedState(
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE,
			desc);

	// Update KvState
	byte[] bytes = new byte[2 * channel.config().getWriteBufferHighWaterMark()];

	byte current = 0;
	for (int i = 0; i < bytes.length; i++) {
		bytes[i] = current++;
	}

	int key = 99812822;
	backend.setCurrentKey(key);
	state.update(bytes);

	// Request
	byte[] serializedKeyAndNamespace = KvStateRequestSerializer.serializeKeyAndNamespace(
			key,
			IntSerializer.INSTANCE,
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE);

	long requestId = Integer.MAX_VALUE + 182828L;

	assertTrue(registryListener.registrationName.equals("vanilla"));

	ByteBuf request = KvStateRequestSerializer.serializeKvStateRequest(
			channel.alloc(),
			requestId,
			registryListener.kvStateId,
			serializedKeyAndNamespace);

	// Write the request and wait for the response
	channel.writeInbound(request);

	Object msg = readInboundBlocking(channel);
	assertTrue("Not ChunkedByteBuf", msg instanceof ChunkedByteBuf);
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:72,代码来源:KvStateServerHandlerTest.java


示例15: runTestDeclineOnCheckpointError

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
private void runTestDeclineOnCheckpointError(AbstractStateBackend backend) throws Exception{

		TestDeclinedCheckpointResponder checkpointResponder = new TestDeclinedCheckpointResponder();

		Task task =
			createTask(new FilterOperator(), backend, checkpointResponder, false);

		// start the task and wait until it is in "restore"
		task.startTaskThread();

		checkpointResponder.declinedLatch.await();

		Assert.assertEquals(ExecutionState.RUNNING, task.getExecutionState());

		task.cancelExecution();
		task.getExecutingThread().join();
	}
 
开发者ID:axbaretto,项目名称:flink,代码行数:18,代码来源:TaskCheckpointingBehaviourTest.java


示例16: createTask

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
private static Task createTask(
	StreamOperator<?> op,
	AbstractStateBackend backend,
	CheckpointResponder checkpointResponder,
	boolean failOnCheckpointErrors) throws IOException {

	Configuration taskConfig = new Configuration();
	StreamConfig cfg = new StreamConfig(taskConfig);
	cfg.setStreamOperator(op);
	cfg.setOperatorID(new OperatorID());
	cfg.setStateBackend(backend);

	ExecutionConfig executionConfig = new ExecutionConfig();
	executionConfig.setFailTaskOnCheckpointError(failOnCheckpointErrors);

	JobInformation jobInformation = new JobInformation(
			new JobID(),
			"test job name",
			new SerializedValue<>(executionConfig),
			new Configuration(),
			Collections.emptyList(),
			Collections.emptyList());

	TaskInformation taskInformation = new TaskInformation(
			new JobVertexID(),
			"test task name",
			1,
			11,
			TestStreamTask.class.getName(),
			taskConfig);

	TaskKvStateRegistry mockKvRegistry = mock(TaskKvStateRegistry.class);
	TaskEventDispatcher taskEventDispatcher = new TaskEventDispatcher();
	NetworkEnvironment network = mock(NetworkEnvironment.class);
	when(network.createKvStateTaskRegistry(any(JobID.class), any(JobVertexID.class))).thenReturn(mockKvRegistry);
	when(network.getTaskEventDispatcher()).thenReturn(taskEventDispatcher);

	BlobCacheService blobService =
		new BlobCacheService(mock(PermanentBlobCache.class), mock(TransientBlobCache.class));

	return new Task(
			jobInformation,
			taskInformation,
			new ExecutionAttemptID(),
			new AllocationID(),
			0,
			0,
			Collections.<ResultPartitionDeploymentDescriptor>emptyList(),
			Collections.<InputGateDeploymentDescriptor>emptyList(),
			0,
			null,
			mock(MemoryManager.class),
			mock(IOManager.class),
			network,
			mock(BroadcastVariableManager.class),
			mock(TaskManagerActions.class),
			mock(InputSplitProvider.class),
			checkpointResponder,
			blobService,
			new BlobLibraryCacheManager(
				blobService.getPermanentBlobService(),
				FlinkUserCodeClassLoaders.ResolveOrder.CHILD_FIRST,
				new String[0]),
			new FileCache(new String[] { EnvironmentInformation.getTemporaryFileDirectory() }),
			new TestingTaskManagerRuntimeInfo(),
			UnregisteredMetricGroups.createUnregisteredTaskMetricGroup(),
			mock(ResultPartitionConsumableNotifier.class),
			mock(PartitionProducerStateChecker.class),
			Executors.directExecutor());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:71,代码来源:TaskCheckpointingBehaviourTest.java


示例17: testQueryUnknownKey

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests the failure response with {@link UnknownKeyOrNamespaceException} as cause
 * on queries for non-existing keys.
 */
@Test
public void testQueryUnknownKey() throws Exception {
	KvStateRegistry registry = new KvStateRegistry();
	AtomicKvStateRequestStats stats = new AtomicKvStateRequestStats();

	MessageSerializer<KvStateInternalRequest, KvStateResponse> serializer =
			new MessageSerializer<>(new KvStateInternalRequest.KvStateInternalRequestDeserializer(), new KvStateResponse.KvStateResponseDeserializer());

	KvStateServerHandler handler = new KvStateServerHandler(testServer, registry, serializer, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	KeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<Integer> desc = new ValueStateDescriptor<>("any", IntSerializer.INSTANCE);
	desc.setQueryable("vanilla");

	backend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, desc);

	byte[] serializedKeyAndNamespace = KvStateSerializer.serializeKeyAndNamespace(
			1238283,
			IntSerializer.INSTANCE,
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE);

	long requestId = Integer.MAX_VALUE + 22982L;

	assertTrue(registryListener.registrationName.equals("vanilla"));

	KvStateInternalRequest request = new KvStateInternalRequest(registryListener.kvStateId, serializedKeyAndNamespace);
	ByteBuf serRequest = MessageSerializer.serializeRequest(channel.alloc(), requestId, request);

	// Write the request and wait for the response
	channel.writeInbound(serRequest);

	ByteBuf buf = (ByteBuf) readInboundBlocking(channel);
	buf.skipBytes(4); // skip frame length

	// Verify the response
	assertEquals(MessageType.REQUEST_FAILURE, MessageSerializer.deserializeHeader(buf));
	RequestFailure response = MessageSerializer.deserializeRequestFailure(buf);

	assertEquals(requestId, response.getRequestId());

	assertTrue("Did not respond with expected failure cause", response.getCause() instanceof UnknownKeyOrNamespaceException);

	assertEquals(1L, stats.getNumRequests());
	assertEquals(1L, stats.getNumFailed());
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:68,代码来源:KvStateServerHandlerTest.java


示例18: testQueryExecutorShutDown

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests the failure response on a rejected execution, because the query executor has been closed.
 */
@Test
public void testQueryExecutorShutDown() throws Throwable {
	KvStateRegistry registry = new KvStateRegistry();
	AtomicKvStateRequestStats stats = new AtomicKvStateRequestStats();

	KvStateServerImpl localTestServer = new KvStateServerImpl(
			InetAddress.getLocalHost(),
			Collections.singletonList(0).iterator(),
			1,
			1,
			new KvStateRegistry(),
			new DisabledKvStateRequestStats());

	localTestServer.start();
	localTestServer.shutdown();
	assertTrue(localTestServer.getQueryExecutor().isTerminated());

	MessageSerializer<KvStateInternalRequest, KvStateResponse> serializer =
			new MessageSerializer<>(new KvStateInternalRequest.KvStateInternalRequestDeserializer(), new KvStateResponse.KvStateResponseDeserializer());

	KvStateServerHandler handler = new KvStateServerHandler(localTestServer, registry, serializer, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	KeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<Integer> desc = new ValueStateDescriptor<>("any", IntSerializer.INSTANCE);
	desc.setQueryable("vanilla");

	backend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, desc);

	assertTrue(registryListener.registrationName.equals("vanilla"));

	KvStateInternalRequest request = new KvStateInternalRequest(registryListener.kvStateId, new byte[0]);
	ByteBuf serRequest = MessageSerializer.serializeRequest(channel.alloc(), 282872L, request);

	// Write the request and wait for the response
	channel.writeInbound(serRequest);

	ByteBuf buf = (ByteBuf) readInboundBlocking(channel);
	buf.skipBytes(4); // skip frame length

	// Verify the response
	assertEquals(MessageType.REQUEST_FAILURE, MessageSerializer.deserializeHeader(buf));
	RequestFailure response = MessageSerializer.deserializeRequestFailure(buf);

	assertTrue(response.getCause().getMessage().contains("RejectedExecutionException"));

	assertEquals(1L, stats.getNumRequests());
	assertEquals(1L, stats.getNumFailed());

	localTestServer.shutdown();
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:71,代码来源:KvStateServerHandlerTest.java


示例19: testChunkedResponse

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests that large responses are chunked.
 */
@Test
public void testChunkedResponse() throws Exception {
	KvStateRegistry registry = new KvStateRegistry();
	KvStateRequestStats stats = new AtomicKvStateRequestStats();

	MessageSerializer<KvStateInternalRequest, KvStateResponse> serializer =
			new MessageSerializer<>(new KvStateInternalRequest.KvStateInternalRequestDeserializer(), new KvStateResponse.KvStateResponseDeserializer());

	KvStateServerHandler handler = new KvStateServerHandler(testServer, registry, serializer, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	AbstractKeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<byte[]> desc = new ValueStateDescriptor<>("any", BytePrimitiveArraySerializer.INSTANCE);
	desc.setQueryable("vanilla");

	ValueState<byte[]> state = backend.getPartitionedState(
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE,
			desc);

	// Update KvState
	byte[] bytes = new byte[2 * channel.config().getWriteBufferHighWaterMark()];

	byte current = 0;
	for (int i = 0; i < bytes.length; i++) {
		bytes[i] = current++;
	}

	int key = 99812822;
	backend.setCurrentKey(key);
	state.update(bytes);

	// Request
	byte[] serializedKeyAndNamespace = KvStateSerializer.serializeKeyAndNamespace(
			key,
			IntSerializer.INSTANCE,
			VoidNamespace.INSTANCE,
			VoidNamespaceSerializer.INSTANCE);

	long requestId = Integer.MAX_VALUE + 182828L;

	assertTrue(registryListener.registrationName.equals("vanilla"));

	KvStateInternalRequest request = new KvStateInternalRequest(registryListener.kvStateId, serializedKeyAndNamespace);
	ByteBuf serRequest = MessageSerializer.serializeRequest(channel.alloc(), requestId, request);

	// Write the request and wait for the response
	channel.writeInbound(serRequest);

	Object msg = readInboundBlocking(channel);
	assertTrue("Not ChunkedByteBuf", msg instanceof ChunkedByteBuf);
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:72,代码来源:KvStateServerHandlerTest.java


示例20: testQueryExecutorShutDown

import org.apache.flink.runtime.state.AbstractStateBackend; //导入依赖的package包/类
/**
 * Tests the failure response on a rejected execution, because the query executor has been closed.
 */
@Test
public void testQueryExecutorShutDown() throws Throwable {
	KvStateRegistry registry = new KvStateRegistry();
	AtomicKvStateRequestStats stats = new AtomicKvStateRequestStats();

	KvStateServerImpl localTestServer = new KvStateServerImpl(
			InetAddress.getLocalHost(),
			Collections.singletonList(0).iterator(),
			1,
			1,
			new KvStateRegistry(),
			new DisabledKvStateRequestStats());

	localTestServer.start();
	localTestServer.shutdown();
	assertTrue(localTestServer.isExecutorShutdown());

	MessageSerializer<KvStateInternalRequest, KvStateResponse> serializer =
			new MessageSerializer<>(new KvStateInternalRequest.KvStateInternalRequestDeserializer(), new KvStateResponse.KvStateResponseDeserializer());

	KvStateServerHandler handler = new KvStateServerHandler(localTestServer, registry, serializer, stats);
	EmbeddedChannel channel = new EmbeddedChannel(getFrameDecoder(), handler);

	int numKeyGroups = 1;
	AbstractStateBackend abstractBackend = new MemoryStateBackend();
	DummyEnvironment dummyEnv = new DummyEnvironment("test", 1, 0);
	dummyEnv.setKvStateRegistry(registry);
	KeyedStateBackend<Integer> backend = abstractBackend.createKeyedStateBackend(
			dummyEnv,
			new JobID(),
			"test_op",
			IntSerializer.INSTANCE,
			numKeyGroups,
			new KeyGroupRange(0, 0),
			registry.createTaskRegistry(dummyEnv.getJobID(), dummyEnv.getJobVertexId()));

	final TestRegistryListener registryListener = new TestRegistryListener();
	registry.registerListener(registryListener);

	// Register state
	ValueStateDescriptor<Integer> desc = new ValueStateDescriptor<>("any", IntSerializer.INSTANCE);
	desc.setQueryable("vanilla");

	backend.getPartitionedState(VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, desc);

	assertTrue(registryListener.registrationName.equals("vanilla"));

	KvStateInternalRequest request = new KvStateInternalRequest(registryListener.kvStateId, new byte[0]);
	ByteBuf serRequest = MessageSerializer.serializeRequest(channel.alloc(), 282872L, request);

	// Write the request and wait for the response
	channel.writeInbound(serRequest);

	ByteBuf buf = (ByteBuf) readInboundBlocking(channel);
	buf.skipBytes(4); // skip frame length

	// Verify the response
	assertEquals(MessageType.REQUEST_FAILURE, MessageSerializer.deserializeHeader(buf));
	RequestFailure response = MessageSerializer.deserializeRequestFailure(buf);

	assertTrue(response.getCause().getMessage().contains("RejectedExecutionException"));

	assertEquals(1L, stats.getNumRequests());
	assertEquals(1L, stats.getNumFailed());

	localTestServer.shutdown();
}
 
开发者ID:axbaretto,项目名称:flink,代码行数:71,代码来源:KvStateServerHandlerTest.java



注:本文中的org.apache.flink.runtime.state.AbstractStateBackend类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java DocumentContent类代码示例发布时间:2022-05-22
下一篇:
Java SAMLContextProvider类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap